When the Optimizer Becomes the Bottleneck: A Close Look at IndexCache

Introduction

Efficiency research in large language models often follows a predictable pattern. We identify a dominant cost, attack it with a clever approximation, and declare victory. Then, sometimes years later, we discover that the auxiliary machinery we introduced to enable the approximation has quietly become the new bottleneck. The paper IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse by Bai, Dong, and colleagues at Tsinghua University and Z.ai is a clean illustration of exactly this dynamic, and the fix it proposes is one of those ideas that feels obvious only after someone has done the careful profiling to expose the problem.

The setting is sparse attention, specifically DeepSeek Sparse Attention (DSA), a production-grade trainable sparse attention mechanism. DSA reduces the quadratic cost of full attention by introducing a lightweight component called a lightning indexer at each layer. The indexer scores all preceding tokens and selects the top-k most relevant ones, so that the expensive core attention only operates over k tokens rather than the full sequence. This brings core attention from O(L²) down to O(Lk). The trouble, and the entire premise of IndexCache, is that the indexer itself still runs at O(L²) at every single layer.

The Hidden Quadratic

The central observation in the paper is delightfully empirical. Per-FLOP, the indexer is cheap relative to the main attention computation. But cheap per-FLOP is not the same as cheap in aggregate. Across N layers, the indexer's total cost is O(NL²), and that term grows quadratically with context length just like the thing it was supposed to eliminate.

The profiling numbers make the stakes concrete. On a 30B DSA model, the indexer's share of total prefill latency climbs from 27% at 10K tokens to 81% at 200K tokens. During decode the growth is gentler, rising from 27% to 41% over the same range, but the prefill figure is striking. At long context, the indexer is no longer a side cost. It is essentially the entire prefill budget. The lightning indexer was designed as a way to skip computation, and at scale it has become the computation.

This is a textbook example of what I would call Amdahl drift. When you optimize the dominant term in a system, the residual terms do not stay residual. They become the new ceiling, and any further speedup depends on attacking them. The interesting part is that the residual here is not some irreducible floor. It is itself a quadratic cost hiding behind a "lightweight" label.

The Redundancy That Was Sitting There

The fix rests on a property that, in retrospect, the field already half knew. Top-k token selections are highly correlated across consecutive layers. Prior work on full attention models had documented this cross-layer token selection stability and exploited it by reusing indices from designated anchor layers. The wrinkle is that in DSA there is no full attention to anchor against; the only place full-sequence scoring happens is inside the indexer itself.

The authors verify the redundancy directly for DSA by computing pairwise top-k overlap across all layers. Adjacent layers share between 70% and 100% of their selected tokens, and the overlap heatmap reveals distinct clusters of layers with mutually high agreement. If layer 12 and layer 13 are selecting nearly the same tokens, running a fresh O(L²) indexer at both is wasteful.

IndexCache turns this observation into a simple architecture. Layers are partitioned into:

• Full (F) layers that retain their own indexers, compute fresh top-k indices, and cache them.
• Shared (S) layers that skip indexer computation entirely and reuse the cached indices from the nearest preceding F layer.

The inference loop change is minimal: a single conditional branch. The cache, importantly, is a temporary buffer that holds only the current index tensor and is overwritten at each F layer, so it adds no GPU memory beyond what DSA already allocates. This is the kind of implementation detail that matters for adoption. A method that promises speedup but quietly inflates the memory footprint at long context is far less attractive.

Two Routes to the Same Configuration

The paper offers two complementary methods to decide which layers keep their indexers, and they reflect two different deployment constraints.

Training-free IndexCache works on any off-the-shelf DSA model with no weight updates. The authors note that naive uniform interleaving, keeping every other indexer, degrades quality. Instead they use a greedy layer selection algorithm that picks which layers retain indexers by directly minimizing language modeling loss on a small calibration set. The greedy solution retains only one quarter of the indexers while matching the original model's downstream performance. The fact that uniform interleaving fails but a data-driven greedy search succeeds is itself informative. It tells us the cross-layer clusters are not evenly spaced. Some regions of the network have more index drift than others, and the optimal pattern has to respect that structure.

Training-aware IndexCache instead optimizes the model parameters for sharing. It introduces a multi-layer distillation loss that trains each retained indexer against the averaged attention distributions of all the layers it serves. Under this objective, even a plain uniform interleaved pattern reaches parity with the per-layer design. The intuition is that if a single indexer must serve several layers, you should train it to be a good consensus predictor for that whole group rather than a specialist for one layer.

The results are consistent across both routes. On a 30B DSA model evaluated over nine long-context and reasoning benchmarks, both methods retain only one quarter of indexers with negligible quality loss, delivering up to 1.82x prefill speedup and 1.48x decode speedup at 200K context. Preliminary experiments on the 744B GLM-5 model remove 50% of indexer computations while maintaining comparable performance, for roughly a 1.2x to 1.3x end-to-end speedup.

My Take

What I find most interesting is not the speedup number but what the two methods reveal about the difference between exploiting redundancy and engineering it. Training-free IndexCache exploits redundancy that already exists, which is why it needs a careful greedy search to find the layers where redundancy is genuine. Training-aware IndexCache manufactures redundancy by teaching shared indexers to be averaging predictors, which is why even a dumb uniform pattern then works. This distinction generalizes. Many "free" efficiency tricks are really just discovering structure the model happened to learn, and they hit a ceiling at whatever structure naturally emerged. Once you are willing to train, you can shape the structure to be more shareable, and the ceiling moves.

There are honest limitations worth flagging. The greedy calibration approach optimizes for LM loss on a fixed calibration set, and it is fair to ask how robust the selected pattern is across very different downstream distributions, especially agentic and tool-use traces that differ sharply from pretraining text. The reported 70% to 100% overlap is also a range, and the bottom of that range matters: a layer that shares only 70% of its selections is throwing away 30% of potentially relevant tokens when it reuses a neighbor's indices. The benchmarks suggest this is tolerable in aggregate, but it could matter for specific tasks where the long tail of attended tokens carries real signal, such as exact retrieval over many distractors.

I would also note that IndexCache and DSA together push toward a layered approximation stack: sparse core attention on top of shared indexing on top of, presumably, future approximations to the indexer's own scoring. Each layer of approximation compounds error, and the field will eventually need a principled account of how these errors interact rather than validating each one independently.

Conclusion

IndexCache is a small architectural change with an outsized payoff, and it lands because the authors did the unglamorous work of profiling where time actually goes at long context. The broader lesson is the one the accompanying summary already gestured at: as we optimize each bottleneck, the auxiliary machinery becomes the new ceiling, and cross-layer redundancy is a rich and underexploited source of efficiency.

The open question I would pose is whether index sharing should be static at all. The overlap heatmap shows clusters, but those clusters may shift with input content or generation phase. A version of IndexCache that decides at runtime whether a layer's neighbor is still a good proxy, refreshing only when drift exceeds a threshold, could capture more of the redundancy without paying for indices that have genuinely changed. The static partition is the right first move. The dynamic one may be where the next factor of speedup hides.