Reasoning Without Training: Why Where You Cut Matters More Than How Much You Search

The dominant story of the last two years of language model progress has been reinforcement learning. Take a capable base model, apply RLHF or RLVR with verifiers and curated datasets, and watch its performance on mathematics, coding, and scientific reasoning climb. This pipeline has produced impressive systems. But it has also produced a quietly unsettling question that the field has not fully answered: when we post-train a model with RL, are we teaching it genuinely new reasoning procedures, or merely reorganizing capabilities it already possessed?

The paper Reasoning with Sampling: Cutting at Decision Points by Felix Zhou, Anay Mehrotra, and Quanquan C. Liu sits squarely on the side that says: mostly the latter. And it makes a practical contribution that follows from taking that position seriously. If post-training mainly concentrates probability mass on good reasoning traces the base model already assigns non-trivial likelihood to, then the real bottleneck is not training at all. It is sampling.

The Sharpening View and the Power Distribution

The conceptual backbone here is what the authors call the sharpening view. A growing body of work (Yue et al., Sun et al., and others cited in the paper) has observed that RL-trained models often improve pass@1 without expanding the set of reasoning paths reachable in the base distribution. In other words, RL converts strong top-k performance into strong top-1 performance. The good answers were already in there; RL just made them more likely to surface on the first try.

If that is true, you can try to get the same effect without any training. Karan and Du formalized this by targeting the power distribution:

$$\Pi_T(x) \propto p(x)^\alpha$$

where $p$ is the base model's distribution over complete reasoning traces and $\alpha > 1$ sharpens it, upweighting traces the base model already considers likely. This is appealing precisely because it is training-free, dataset-free, and verifier-free. It is pure test-time scaling using nothing but the base model's own probabilities.

The catch is that sampling from $\Pi_T$ is hard. You cannot just autoregressively decode from it, because the normalization constant couples the whole sequence. Karan and Du used a stagewise Metropolis-Hastings (MH) sampler: pick a cut position in the current trace, keep the prefix, resample the suffix from a proposal model, then accept or reject via the MH correction that guarantees the chain converges to $\Pi_T$. It works. But its efficiency, specifically how fast the Markov chain mixes, is the bottleneck for the entire approach.

The Insight: Not All Cuts Are Equal

This is where the paper makes its move, and it is a clean one. For any MCMC method to be useful, it must be able to move between the modes of its target distribution. For a distribution over token sequences, the modes correspond to qualitatively different lines of reasoning. A proof by induction and a proof by contradiction might both have high probability under the sharpened distribution, but to get from one to the other, the sampler has to revisit the early decision where the proof strategy was chosen.

Here is the problem with the prior approach. If you choose the cut position uniformly at random over the tokens in a trace, you almost always land in the middle of some local computation. Cutting inside $\sqrt{0^3+3^3}$ and resampling the suffix just rewrites a few nearby tokens. It does not reconsider whether you should have computed $r = \sqrt{x^3+y^3}$ in the first place rather than $\sqrt{x^2+y^2}$. The chain wastes its effort polishing arithmetic instead of revisiting consequential decisions.

The authors' fix is Entropy-Cut Metropolis-Hastings. The observation is that key decision points tend to coincide with high next-token entropy: when the model is genuinely uncertain about how to proceed (which strategy, which algorithm, which lemma), the next-token distribution is flat. Routine continuations, by contrast, are low-entropy. So instead of cutting uniformly, Entropy-Cut places cuts preferentially at high-entropy positions. The MH correction is adjusted accordingly, so the target distribution remains exactly $\Pi_T$. You change the proposal mechanism, not the thing you are sampling from.

What I find compelling is that the authors do not just assert this heuristic. They back it with a stylized reasoning-tree model, where valid traces are root-to-leaf paths and branch nodes are decision points. In this model they prove (Theorem 4.1) that entropy-cut mixing time scales with the number of decisions $k$, while uniform-cut mixing can scale with the token depth $T$. Since $T$ can be vastly larger than $k$ (a proof might have thousands of tokens but only a handful of genuine branch points), this is potentially an enormous gap. Entropy serves as the bridge between the abstract tree model and the messy reality of token sequences: branch nodes are where entropy spikes.

Results and What They Tell Us

Empirically, the paper evaluates across Qwen2.5-7B, Qwen2.5-Math-7B, Qwen3-8B-Base, Phi-3.5-mini-instruct, and Phi-4-mini-instruct on MATH500, HumanEval, GPQA Diamond, and AIME26. On Qwen2.5-7B, Entropy-Cut reaches 71.9 percent on MATH500, 68.9 percent on HumanEval, and 30.2 percent on GPQA Diamond, consistently beating standard sampling, low-temperature sampling, SMC, TMC, and the uniform-cut MH baseline. The margins over uniform-cut are modest but consistent, which is what you would expect from a better proposal distribution rather than a different target.

Two details deserve emphasis. First, this beats RL-trained models on these benchmarks with zero training, no curated data, and no verifier. That is a strong statement in favor of the sharpening view. Second, the authors report that, across multiple samples, they avoid a collapse in diversity despite improved single-shot performance. This matters because naive sharpening tends to crush diversity, which is exactly what you do not want if you intend to combine inference-time search with later sampling or ensembling.

My Take

I think the most important thing in this paper is not the specific algorithm but what it implies about where effort should go. If the sharpening view is even approximately correct, then a large fraction of the compute the field pours into RL post-training is buying us something we could partially recover at inference time with a better sampler. That reframes the economics. Training is a fixed, expensive, centralized cost. Inference-time search is flexible and can be dialed up per query.

There is also a deeper methodological point here that I find underappreciated. The entropy-cut idea is essentially importance sampling guided by the model's own uncertainty signal. Next-token entropy is a free, internal estimate of where the model is making a decision versus executing a routine. We have seen related ideas elsewhere, for instance entropy-based exploration in RL and uncertainty-guided decoding, but using it specifically to direct MCMC proposals toward decision points is an elegant unification of two literatures.

The honest limitations are real, though. The mixing-time guarantee lives in a stylized tree model; real traces do not have clean branch nodes, and entropy is a noisy proxy. Some high-entropy positions are genuinely inconsequential (formatting tokens, synonym choices), and some consequential decisions might be low-entropy if the base model is confidently wrong. The method also inherits the fundamental ceiling of the sharpening view: if a reasoning path has essentially zero probability under the base model, no amount of sharpening will find it. You cannot sharpen what is not there. This is precisely where RL with verifiers might still teach something new, by reinforcing rare correct paths until they become probable.

Open Questions

The result that stays with me is the diversity preservation. If we can sharpen single-shot accuracy without collapsing the distribution, then power sampling composes naturally with downstream search and self-consistency, which suggests a layered inference stack rather than a single trick. I would want to see how entropy-cut behaves on much longer traces, where the gap between $k$ and $T$ is largest and the payoff should be greatest, and whether learned cut-position predictors outperform raw entropy.

The broader question is whether the future of reasoning is smarter inference-time search over fixed base distributions, with training reserved for genuinely expanding the support of what the model can do. This paper is a strong data point that the sampling side has been underexplored. The bottleneck, as the authors argue, may not be training at all.