Zeroth-Order Fine-Tuning Was an Inference Workload All Along

When we think about fine-tuning a large language model, we almost reflexively picture a training loop: forward pass, backward pass, gradient accumulation, optimizer step. The mental model is so ingrained that even methods explicitly designed to avoid backpropagation get implemented inside the same scaffolding. A recent paper, "LLM Zeroth-Order Fine-Tuning is an Inference Workload" by Zelin Li and Caiwen Ding, makes the argument that this is a category error. Zeroth-order (ZO) fine-tuning is not training with the gradients removed. It is a structured stream of inference-style scoring requests that we have been forcing into the wrong execution substrate.

The claim is sharp and, once stated, almost obvious. But the consequences are substantial: by routing ZO fine-tuning through a serving runtime like vLLM, the authors report an 8.13x training-time speedup on OPT-13B SST-2, completing a 20k-step LoZO run in 0.51 estimated training hours versus 4.15 hours for the official baseline, while preserving accuracy at 0.922 (and 0.931 on full validation).

Why ZO Fine-Tuning Is Really Scoring

A two-point ZO estimator works by perturbing parameters in a random direction and comparing the resulting objective values. Given parameters θ and a sampled direction z, you compute:

• L+ = L(θ + εz)
• L- = L(θ - εz)

and form the scalar coefficient c = (L+ - L-) / 2ε. The parameter update is then θ ← θ - ηcz. MeZO popularized this forward-only view for LLMs, and LoZO refined it by making the perturbation direction matrix-wise and low-rank, sampling factors U and V with rank r much smaller than the layer dimensions.

The key structural observation is about where the cost lives. The update itself (writing the small low-rank delta) is cheap. The expensive operation is the pair of forward objective evaluations. And those two evaluations are not arbitrary forward passes. They are two closely related scoring requests over nearly identical parameter states, differing only by a small perturbation εz. This is precisely the kind of workload a modern serving runtime is built to exploit: batched forward execution, token log-probability extraction, resident LoRA adapters, and CUDA graph capture.

The mismatch the paper identifies is that a conventional Python training loop hides all of this from the runtime. Each forward call appears as an isolated, fragmented step. The optimizer is asking for structured inference; the system delivers a sequence of disconnected training-loop invocations. The runtime never gets to see, batch, or schedule the operations that actually dominate the step.

The System Reorganization

What I find compelling about this work is the discipline in its scope. The authors do not propose a new optimizer. The ZO objective, the perturbation rule, the random-direction stream, and the update rule all remain unchanged. They preserve the optimizer semantics exactly, including a careful note that the released LoZO implementation does not apply the explicit 1/r division that appears in the paper's estimator, and they match the implementation they compare against. This matters for honest benchmarking: a speedup claim is only meaningful if the optimization trajectory is preserved.

The system contributions are three concrete execution-path changes:

• Direct worker scoring. Objective evaluations are issued through a vLLM direct-worker path rather than through the high-level request and trainer-loop machinery. When all you need is option-token losses, the framing overhead of a general serving API is pure waste.

• Direct update path. After the ZO coefficient is computed, the low-rank update is applied worker-side, with packed QKV updates batched to cut update overhead.

• Memory-write reduction. This is the most elegant piece. In LoZO's lazy-sampling variant, the V factor is held fixed across a ν-step window while U is resampled each step. The cumulative update therefore stays inside the same low-rank subspace, which means you can accumulate updates directly on the U factor:

  U_{t+1} = U_t - ηc_t G_t, ΔW = U_{t+1} V^T

  Over m steps with the same V, the delta is just (U_t - η Σ c_s G_s) V^T. The runtime writes a compact U accumulator instead of repeatedly materializing full-weight updates. The savings here compound over a long run.

The empirical results extend beyond the headline number. Core-step scaling experiments across OPT-1.3B to OPT-13B yield 2.34x to 7.72x speedups, indicating the benefit is not an artifact of a single model size. A MeZO-style high-rank factorized experiment shows the same runtime paradigm tracking a MeZO-like loss trajectory while running up to 2.55x faster, suggesting the approach transfers across ZO variants rather than being a LoZO-specific trick.

My Take: The Real Story Is the Abstraction

The benchmark numbers are persuasive, but I think the more durable contribution is conceptual. The paper reframes the execution boundary rather than the algorithm. For years we have asked whether serving runtimes can be repurposed to accelerate training loops. This work inverts the question: it asks whether the training loop is simply the wrong substrate for a workload that is dominated by repeated forward scoring.

That inversion has a payoff that the paper gestures toward but does not fully develop: representing accumulated ZO updates and temporary perturbations as dynamically composed LoRA adapter states opens a path to inference-time training. If a fine-tuning step is just an inference-shaped scoring request plus a cheap adapter-state edit, then adaptation can be scheduled like any other inference job. You no longer need a separate training cluster and a separate serving cluster with a model-handoff in between. A serving system that already manages LoRA residency could, in principle, host lightweight continual adaptation as a first-class workload alongside ordinary inference traffic.

This connects to a broader trend. The infrastructure investment in inference serving has dramatically outpaced investment in training-loop tooling for the long tail of fine-tuning jobs. PagedAttention, continuous batching, and CUDA graph capture were all built for throughput-oriented serving. The insight here is that ZO fine-tuning can free-ride on every one of those optimizations because its dominant primitive is, structurally, the same operation serving was built around.

Limitations and Open Questions

Several caveats deserve honesty. First, the evaluation is concentrated on the OPT family and on SST-2, a binary sentiment task with short option-token completions. ZO methods are known to be sensitive to task structure, and SST-2's two-option scoring is an especially favorable case for option-token loss extraction. How the runtime advantage holds on generative or long-context tasks, where the scoring is not a tidy classification over a few tokens, remains an open question.

Second, the speedups are reported as "estimated training hours" for the main long-run result. Estimation is reasonable for very long runs, but the gap between estimated and wall-clock numbers under real scheduling contention deserves scrutiny.

Third, the closest prior work, MobiZO, also connected ZO fine-tuning to inference engines, though targeting on-device and edge settings. The present work targets server-side fine-tuning, so the contributions are complementary rather than competing. Still, it would strengthen the case to see how the two execution paths differ quantitatively where they overlap.

Finally, ZO methods carry an inherent statistical cost: gradient estimates from a handful of random directions are noisy, which is why ZO runs often need many more steps than first-order fine-tuning. A per-step speedup is valuable, but the more important question is total time to a target accuracy. The 20k-step result at 0.922 accuracy is encouraging, but a systematic comparison against first-order LoRA on wall-clock-to-accuracy would clarify when ZO-on-a-serving-runtime is the right choice rather than merely a faster way to run a method that needs many steps.

The most interesting direction this opens, in my view, is whether the "inference-time training" framing can support genuinely online adaptation: a deployed model that absorbs lightweight ZO updates from live feedback, scheduled within the same serving pipeline that answers queries. If that works at scale, the line between training and serving infrastructure starts to dissolve, and the right way to think about fine-tuning becomes a question of scheduling rather than a separate job entirely.