Tracing Motion Instead of Painting Pixels: Sparse Trajectory Prediction for World Models

When we imagine how a scene might unfold, we do not hallucinate every pixel of every intervening frame. We picture a few salient things moving: the ball rolling toward the pocket, the cup tipping, the person stepping off the curb. The rest of the visual field, the texture of the felt, the grain of the wood, the color of the wall, is irrelevant to the prediction. This intuition sits at the center of Envisioning the Future, One Step at a Time by Baumann, Wiese, Martorella, Kalayeh, and Ommer, a paper that reframes future prediction as step-wise inference over sparse point trajectories rather than dense video or latent-space generation.

The proposal is deceptively simple, but it cuts against a strong current in the world-model literature. I want to unpack why the design choices matter, what the benchmark contributes, and where I think the approach will run into friction.

The visual tax and why it is expensive

The authors introduce a useful piece of vocabulary: the visual tax. A model incurs the visual tax when it must generate appearance, every pixel of every frame, before it can reason about motion. Video diffusion models and most latent-space world models pay this tax in full. They spend enormous representational and computational capacity rendering the look of a scene when the actual quantity of interest is how things move.

This is not a minor inefficiency. Consider what dense prediction implies for counterfactual exploration. If you want to evaluate thousands of candidate billiard shots, as in the paper's running example, a dense video model has to synthesize thousands of full video rollouts, each one a high-dimensional generation problem. The cost scales with appearance fidelity, frame rate, and resolution, none of which contribute to answering "does the orange ball reach the target pocket?"

The paper's alternative is to model the world through the trajectories of a sparse set of tracked points. Once you commit to predicting only where points go, the dimensionality of the problem collapses. You are no longer painting; you are tracing. This is the conceptual move that the related tweet compresses into a slogan, and it is the right one to emphasize because everything else in the method follows from it.

Autoregressive diffusion over short, locally predictable transitions

The second design decision is equally important and, I think, underappreciated in discussions of world models. The authors do not predict the distant future in a single leap. They formulate prediction as an autoregressive diffusion model that advances trajectories through short transitions, with each step conditioned on the previous one.

Why does this matter? The paper argues, correctly in my view, that single-shot long-horizon prediction imposes an impossible burden. To predict where a billiard ball will be after several collisions in one step, a model must implicitly account for every intervening contact simultaneously. The space of outcomes branches combinatorially. Unless the task is trivially "one-hop" or the model has unbounded capacity, the only tractable route is to unfold the future incrementally, collision by collision, where the local web of interactions remains manageable.

There is a deeper statistical justification here. Over a short horizon, the distribution of next positions is locally predictable and often close to unimodal or low-entropy. Uncertainty is small and easy to model. By chaining these short steps autoregressively, the model lets uncertainty grow naturally over time, which is exactly how real stochastic systems behave. Multi-modality emerges from the accumulation of small, well-calibrated branchings rather than from one heroic act of distributional modeling. The diffusion component handles the stochasticity at each step; the autoregressive structure handles the temporal compounding.

This is a clean separation of concerns. It also explains why the method can generate diverse futures rather than collapsing to a single mean trajectory, a chronic failure mode of regression-based motion predictors trained with squared-error losses.

OWM: benchmarking open-set motion

A method like this needs an evaluation that rewards the right things. Existing motion-prediction benchmarks are often domain-specific, for instance autonomous driving trajectories or constrained physics simulations, where the set of agents and dynamics is known in advance. The authors introduce OWM, a benchmark for open-world motion prediction built on diverse in-the-wild videos.

The key property OWM targets is the evaluation of trajectory distributions under real-world uncertainty, not just point estimates. This is the correct framing. If a model is meant to envision many plausible futures, scoring it against a single ground-truth continuation penalizes exactly the diversity you want. A proper benchmark has to assess both accuracy (do predicted trajectories stay physically plausible and consistent with what actually happens) and variability (does the predicted distribution cover the genuine range of outcomes). Getting this balance into a metric is hard, and the field has historically leaned too heavily on accuracy alone.

On their evaluations, the authors report matching or surpassing dense simulators in predictive accuracy while sampling orders of magnitude faster. The speed claim is the load-bearing result. If you can roll out thousands of futures at the cost dense models pay for one, then planning-by-simulation becomes practical in open-set settings, which is the regime where it has been out of reach.

My take: what this enables and what it leaves open

I find the sparse-dynamics framing convincing, and I want to push on its implications.

First, the most interesting capability is not prediction per se but planning through counterfactual rollout. The billiard example in Figure 1 is illustrative: by exploring directly in motion space and optionally conditioning on an initial motion cue (the cue strike), the model can search over candidate actions and select one that reaches a goal. This is model-predictive control with a learned, open-set, stochastic simulator. The speed advantage is what makes the search tractable. I expect the most valuable downstream applications to be in this planning-and-selection loop rather than in passive forecasting.

Second, there is an honest limitation in the sparse representation itself. Points are not objects. A trajectory model that does not render appearance also does not natively know about occlusion, object birth and death, or interactions with entities that were never tracked because they were not visible in the initial frame. The abstraction that makes the method efficient, "trace only the changes that matter," presupposes that the important points are identified up front. In genuinely open scenes, salience is not given; a parked car becomes important only when it starts to move. How the model handles the appearance of new dynamic content from a single initial image is, to me, the central open question.

Third, the reliance on point tracking inherits the limits of upstream trackers. Errors in tracking become errors in the supervision signal. Fast motion, motion blur, and thin structures are notoriously difficult to track, and these are often precisely the cases where dynamics are most interesting.

Connecting to the broader field, this work sits alongside a growing recognition that world models do not need to be pixel-perfect to be useful for control. Latent-action models and object-centric predictors share the instinct to discard appearance. What this paper adds is a concrete, scalable instantiation grounded in point trajectories plus a sober argument for why step-wise beats single-shot for long horizons.

Conclusion

The thesis is that we do not paint the future, we trace the few changes that matter, and the engineering follows faithfully from it: sparse trajectories to avoid the visual tax, autoregressive diffusion to let uncertainty grow over short, locally predictable steps, and OWM to evaluate distributions rather than points. The result is a model that matches dense simulators on accuracy while sampling fast enough to make open-set planning realistic.

The open questions I would most like to see addressed are how salience is bootstrapped when important content is initially static or invisible, how robust the method is to tracker noise, and whether the learned dynamics generalize to genuinely novel interaction types rather than interpolating familiar ones. If those hold up, tracing motion instead of painting pixels may become the default substrate for world models built for action rather than spectacle.