Rethinking Expert Allocation in MoE: Why Every Layer Should Not Own Its Own Experts

Introduction

Mixture-of-Experts (MoE) architectures have become the standard tool for scaling language models beyond the point where dense computation remains tractable. The appeal is straightforward: route each token to a small subset of expert feed-forward networks, growing total parameter count while keeping per-token compute roughly constant. Models like Switch Transformer, GShard, and a long line of production systems have followed essentially the same design rule. That rule, rarely questioned, is that each transformer layer owns its own private set of experts, selected by a layer-specific router.

A recent preprint, UniPool: A Globally Shared Expert Pool for Mixture-of-Experts (Huang et al., 2026), argues that this convention is more habit than necessity. The paper makes a sharper claim: per-layer expert ownership is not just a convenience but a source of redundancy and wasted parameters, and we can do better by treating expert capacity as a single global budget shared across depth. I want to walk through the evidence they present, the mechanics of their proposed architecture, and what I think the broader implications are for how we scale sparse models.

The Probe That Exposes Redundancy

The most persuasive part of the paper is not the new architecture but the diagnostic that motivates it. The authors run a routing-randomization probe on three production MoE models. The experiment is simple: take a single MoE layer in the deeper half of the network, throw away its learned top-k router, and replace it with uniform random routing. If the learned router were doing critical, layer-specific work, accuracy should collapse. Instead, downstream accuracy drops by only 1.0 to 1.6 points.

This is a striking result, and it deserves careful interpretation. A drop of one to two points is not zero, so the routers are doing something. But the magnitude is small enough to suggest that the experts within a deep layer are largely interchangeable from the perspective of any given token. If random assignment among them costs so little, then the experts are learning highly similar transformations, and the careful per-layer routing is mostly redundant.

The natural inference, which the paper makes explicit, is that standard MoE training duplicates expert functions across layer-private budgets. Each layer independently learns its own approximations of similar transformations because it has no mechanism to reuse what another layer already learned. From a parameter-efficiency standpoint, this is exactly the kind of waste we should want to eliminate. The linear coupling between depth and expert parameters means that as you add layers, you keep paying for capacity that may already exist elsewhere in the network.

I find this framing convincing, though I would have liked the probe extended to early layers as well. The paper focuses on the deeper half, and there is reason to believe early layers, which sit closer to token-level features, may show more genuine routing specialization. The redundancy story may be depth-dependent in ways that matter for how aggressively you can share.

The UniPool Architecture

Given the diagnosis, the prescription is intuitive: stop giving each layer its own experts, and instead maintain a single global expert pool that every layer's router can draw from. Crucially, UniPool keeps independent per-layer routers. The sharing happens at the level of expert parameters, not routing decisions. Layer 3 and layer 12 can both select expert 7 if it is useful to them, but each retains its own logic for deciding what to select.

This sounds clean, but it introduces two non-trivial problems that the paper addresses directly.

Load balancing becomes a global question. In standard MoE, auxiliary load-balancing losses are applied per layer to prevent dead experts: if a layer-private expert receives no tokens, its parameters are wasted. Under a shared pool, this layer-local notion of deadness no longer matches where parameters live. An expert ignored by layer 3 might be heavily used by layer 8, and that is fine. Forcing every layer to use every expert would directly contradict the goal of layer-specific routing and cross-layer specialization. UniPool introduces a pool-level auxiliary loss that aggregates token-to-expert assignments across all layers and balances utilization over the global pool. The objective prevents globally dead experts while allowing each layer to specialize on a subset.

This is the right design decision, and it is worth emphasizing why. The auxiliary loss is not a cosmetic detail; it defines what "balanced" even means in a shared system. Getting it wrong would either collapse the pool into uniform usage, destroying specialization, or allow large portions of the pool to atrophy.

Routing stability across depth. Standard softmax gating is designed for layer-specific experts and can produce layer-dependent logit scales. When routers at different depths all select from the same large pool, inconsistent logit magnitudes translate into inconsistent routing sharpness and unstable competition among shared experts. UniPool adopts NormRouter, which replaces softmax with an L2-normalize-then-ReLU scoring function plus a learnable scaling factor. Normalization removes sensitivity to layer-specific hidden-state scale, ReLU induces sparse competition over the large pool, and the learnable scale lets each router tune its routing strength. This is a sensible co-design: a shared pool stresses the router in ways that softmax was never meant to handle.

The Results and What Pool Size Really Means

The experiments span five LLaMA-architecture scales (182M, 469M, 650M, 830M, and 978M parameters), each trained on 30B tokens from the Pile. UniPool consistently improves validation loss and perplexity over matched vanilla MoE baselines, with validation loss reductions of up to 0.0386.

The more interesting finding is about parameter efficiency. Reduced-pool UniPool variants using only 41.6% to 66.7% of the vanilla expert-parameter budget match or outperform layer-wise MoE. This reframes pool size as an explicit depth-scaling hyperparameter. Under a shared design, expert parameters need not grow linearly with depth; they can grow sublinearly while remaining more efficient. The paper also notes that UniPool's gains compose with finer-grained expert decomposition, suggesting the technique stacks with existing efficiency tricks rather than competing with them.

A loss reduction of 0.0386 is modest in absolute terms, and I want to be honest about that. The headline result is not raw quality but the decoupling of depth from parameter count. The ability to reach parity on roughly half the expert budget is the practical payoff, because it changes the cost curve of scaling depth.

My Take

What I find most valuable here is conceptual rather than numerical. UniPool reframes the unit of capacity in MoE from "experts per layer" to "experts in the network." This is a more honest accounting of what the model is actually learning. If transformations recur across depth, and the probe strongly suggests they do, then forcing the model to relearn them per layer is a tax we have been paying without justification.

There are limitations worth flagging. The largest model tested is under 1B parameters, and the redundancy story may shift at the scales where MoE matters most, in the tens or hundreds of billions of parameters. Sharing introduces routing contention and potential interference; an expert optimized for two very different layers may be pulled toward a compromise that serves neither well. The pool-level balance loss mitigates this, but interference is a different failure mode than dead experts, and it is harder to measure. There are also serious systems questions. A global pool complicates expert-parallel sharding, since experts can no longer be cleanly partitioned by layer across devices. The paper's gains are measured in parameters and loss, not in wall-clock training or inference throughput on distributed hardware.

I would also push on the random-routing probe as evidence. It demonstrates that experts within a layer are interchangeable, which supports intra-layer redundancy. The leap to cross-layer sharing is plausible but is a stronger claim that the architecture's success then validates after the fact.

Conclusion

UniPool questions an assumption so embedded in MoE practice that few have tested it: that each layer needs its own experts. The routing-randomization probe provides a clean, falsifiable argument that the assumption is wasteful, and the shared-pool architecture, with its pool-level balancing and NormRouter, turns that observation into a working system that reaches parity on a fraction of the expert budget.

The open questions are about scale and systems. Does the redundancy hold at frontier model sizes? Can a global pool be sharded efficiently across hundreds of accelerators? And how far can pool size be pushed down before specialization genuinely suffers? If these answers hold up, the field may need to retire the per-layer expert convention and start treating expert capacity as what it arguably always was: a global resource to be allocated where it does the most good.