A new architecture for Mixture-of-Experts (MoE) models, called UniPool, proposes a significant departure from current design principles by consolidating expert capacity into a single, shared pool rather than allocating separate experts to each layer. This approach, detailed in a paper on arXiv, challenges the assumption that each transformer layer requires its own distinct set of experts.
Modern MoE models typically assign a unique set of experts to each transformer layer. This convention links the growth of model depth directly to a linear increase in expert parameters. However, recent analyses suggest this structure may be inefficient. Experiments indicate that replacing a deeper layer's learned router with a random routing mechanism only causes a minor drop in accuracy, between 1.0% and 1.6%, across various production MoE models. This observation of redundancy motivated the development of UniPool.
UniPool treats expert capacity as a global budget. Instead of each layer owning its experts, the architecture uses a single shared pool accessible by independent per-layer routers. To ensure stable and balanced training within this shared pool, the researchers introduced a pool-level auxiliary loss. This loss function helps to balance expert utilization across the entire pool. The system also adopts NormRouter for sparse and scale-stable routing into the shared expert pool.
The researchers evaluated UniPool across five different scales of LLaMA-architecture models, ranging from 182 million to 978 million parameters. These models were trained on 30 billion tokens from the Pile dataset. In these experiments, UniPool consistently demonstrated improvements in validation loss and perplexity compared to standard MoE baselines. Across the tested scales, UniPool reduced validation loss by up to 0.0386 relative to vanilla MoE.
Furthermore, the study identified pool size as an explicit hyperparameter for depth scaling. Variants of UniPool using only 41.6% to 66.7% of the expert-parameter budget of traditional MoE models matched or outperformed their layer-wise counterparts at the tested scales. This suggests that with a shared-pool design, expert parameters do not need to scale linearly with model depth. Instead, they can grow sublinearly while maintaining or improving efficiency and effectiveness over traditional MoE models. The benefits of UniPool also appear to combine well with finer-grained expert decomposition.
Mixture-of-Experts (MoE) architectures have become a standard method for scaling large language models (LLMs), allowing for substantial parameter growth while keeping per-token computation nearly constant. This efficiency stems from MoE's ability to activate only the necessary portions of the network for a given input, rather than engaging all parameters for every task. This conditional computation allows models to increase their capacity without a proportional increase in computational cost.
Traditional MoE models, while efficient, can suffer from issues like under-trained experts due to uneven load balancing. The UniPool architecture, by centralizing experts and introducing a pool-level balancing loss, aims to mitigate these problems. The open-sourced code for UniPool is available on GitHub.
The findings suggest that the current per-layer allocation of experts in MoE models may contain redundant capacity. UniPool's approach of a shared expert pool, combined with a pool-level balancing mechanism, offers a path toward more streamlined and efficient large-scale AI models.