Thermodynamic Response Theory as an Operational Framework for Singular Bayesian Models

Singular statistical models have long presented a conceptual tension in Bayesian statistics. While mixtures, neural networks, and low-rank factorizations dominate modern machine learning practice, their theoretical analysis falls outside the scope of classical asymptotics. When the Fisher information matrix degenerates and parameters become non-identifiable, the posterior no longer concentrates on isolated points but rather on complex algebraic varieties. This geometric complexity has made the field's central invariants, such as the real log canonical threshold (RLCT), feel abstract and operationally distant from the daily practice of model selection. In "Thermodynamic Response Functions in Singular Bayesian Models," Sean Plummer proposes a structural reconciliation. By treating posterior tempering as a physical process rather than a computational trick, Plummer demonstrates that widely used criteria like WAIC and singular fluctuation emerge naturally as thermodynamic response functions. This framework does not merely reinterpret existing mathematics; it provides a measurable pathway to access the geometric invariants that have previously remained locked within algebraic geometry.

The Tempering Deformation and Universal Response Identities

The foundation of Plummer's argument rests on a simple but profound observation about posterior tempering. Consider the tempered posterior distribution defined by π_β(θ|D) ∝ π(θ)p(D|θ)^β, where β serves as an inverse temperature parameter interpolating between the prior at β → 0 and the standard posterior at β = 1. While tempering has historically served computational purposes in methods like parallel tempering or annealed importance sampling, Plummer treats it as a continuous deformation of the statistical manifold itself.

This deformation induces a calculus of response functions through what Plummer terms the universal covariance identity. For any observable f(θ), the derivative of its tempered expectation with respect to β equals the posterior covariance between the observable and the log-likelihood ℓ(θ) := log p(D|θ):

$$\frac{d}{d\beta} \mathbb{E}_\beta[f] = \text{Cov}_\beta(f, \ell)$$

This identity transforms the abstract geometry of the posterior into measurable fluctuations. In physical terms, β becomes the control parameter, the log-likelihood plays the role of energy, and covariances become susceptibilities. The derivative structure implies that how much an observable changes under tempering depends entirely on how it fluctuates with the likelihood energy across the posterior landscape.

For singular models, this is particularly consequential. In regular models, the Fisher information is non-degenerate and the posterior concentrates narrowly, making response functions relatively uninteresting. In singular models, however, the posterior mass spreads across non-identifiable directions, creating nontrivial covariance structures. The tempering path must navigate through symmetry breaking, rank collapses, and neural network weight space redundancies. The universal identity shows that these geometric navigations manifest as peaks and transitions in response functions.

Operationalizing Algebraic Invariants Through Response Functions

Perhaps the most significant contribution of Plummer's work lies in connecting these thermodynamic responses to the invariants of singular learning theory. The RLCT, which governs the asymptotic behavior of the marginal likelihood as p(D) ≈ n^{-λ} (log n)^{m-1} / n, has traditionally required complex resolution of singularities to compute. Plummer shows that λ governs the leading slope of the free energy F(β) = -log Z(β) as β → ∞, where Z(β) is the tempered partition function. This transforms an algebraic calculation into a potentially measurable thermodynamic property.

Similarly, singular fluctuation, previously defined through complex asymptotic expansions, acquires a transparent interpretation as the curvature of the tempered free energy at β = 1. The widely applicable information criterion (WAIC), which practitioners use for model comparison without understanding its geometric basis, emerges as a specific fluctuation measure at unit temperature. Plummer demonstrates that WAIC complexity measures predictive fluctuation, placing it within the broader hierarchy of response functions generated by the tempering deformation.

To make these connections rigorous, Plummer formalizes an observable algebra that quotients out functions constant along non-identifiable directions. This algebraic construction addresses a fundamental problem in singular models: many parameter directions do not affect the predictive distribution. By constructing observables that are invariant to these redundancies, Plummer defines true order parameters that collapse when the posterior undergoes structural reorganization. This quotienting step is essential; without it, one would measure parameterization artifacts rather than statistical structure.

Empirical Phase Transitions in Canonical Models

The paper validates this framework through three canonical singular settings: symmetric Gaussian mixtures, reduced-rank regression, and overparameterized neural networks. Across these diverse architectures, Plummer observes strikingly consistent phase-transition-like behavior under tempering. As β increases, order parameters constructed from the observable algebra exhibit sharp collapses, while susceptibilities (the variances and covariances defined by the response identity) display pronounced peaks at specific inverse temperatures.

In the symmetric Gaussian mixture, the tempering path witnesses symmetry breaking as β increases, with the response functions localizing exactly where the posterior shifts from unimodal to multimodal structure. For reduced-rank regression, the response functions detect the algebraic singularity where the effective rank of the coefficient matrix drops. Most intriguingly, in overparameterized neural networks, the susceptibility peaks align with what appear to be structural reorganization events in the posterior geometry, suggesting that the tempering path traverses distinct "phases" of representational complexity.

These empirical results suggest that the RLCT and singular fluctuation are not merely asymptotic abstractions but measurable quantities that manifest as critical points in tempering trajectories. The alignment between complexity measures and structural reorganization indicates that thermodynamic response theory provides a natural language for describing how singular models "learn" as data influence increases.

From Analogy to Experimental Framework

Plummer's framework represents more than a metaphorical mapping between statistical physics and Bayesian statistics. It constitutes a methodological shift in how we conceptualize model complexity. Rather than viewing singular learning theory as a branch of algebraic geometry that happens to describe statistical models, the thermodynamic perspective treats Bayesian inference as an experimental physical system. The statistician becomes an experimental physicist, varying temperature and measuring responses to probe the underlying geometry.

This perspective suggests several avenues for future research. First, the connection to renormalization group flows in physics appears ripe for formalization. If tempering represents a coarse-graining of the likelihood landscape, then the fixed points of this flow might correspond to the "true" model complexity, divorced from parameterization details. Second, the operationalization of RLCT through response functions may finally enable practical estimation in high-dimensional settings, where algebraic methods fail. Estimators based on measuring free energy slopes during tempering could supplement or replace existing approximation schemes.

However, significant limitations remain. The computational cost of exploring the full tempering path scales with the cost of sampling at multiple temperatures, which may prove prohibitive for large-scale neural networks. Additionally, while the observable algebra theoretically quotients out non-identifiable directions, constructing these invariants explicitly remains challenging in practice for complex architectures like deep networks. The paper demonstrates the concept empirically, but general algorithms for automatic quotienting require further development.

Conclusion

Plummer's work establishes thermodynamic response theory as a natural organizing framework for singular Bayesian learning. By showing that WAIC, singular fluctuation, and the RLCT emerge from a unified hierarchy of tempered expectations, the paper bridges the gap between the abstract geometry of singular models and the operational practice of model selection. The observation of phase-transition-like behavior across mixtures, regression models, and neural networks suggests that these phenomena are universal features of singular learning, not artifacts of specific parameterizations.

Looking forward, the most pressing question concerns scaling. Can response function estimators maintain precision as models grow to billions of parameters? If so, we may eventually treat model selection as a genuine experimental discipline, probing architectures with temperature variations to map their geometric and predictive structure. Regardless of practical implementation, the conceptual contribution is clear. Singular models are not pathological exceptions but thermodynamic systems with rich response structures, and understanding them requires the full conceptual toolkit of statistical physics.