When the Judge Hallucinates: On the Reliability of Learned 3D Consistency Metrics

The Problem With Asking a Reconstructor to Be a Referee

There is a quiet assumption baked into nearly every multiview 3D pipeline: that the images we feed in are genuine observations of one static scene. We assume the photographs in a sparse-view reconstruction set, or the frames generated by a novel view synthesis (NVS) model, actually depict the same physical world from different angles. Under that assumption, asking whether a reconstruction is "3D consistent" is a sensible question.

But the assumption breaks constantly in practice. Internet-scraped images contain outliers and duplicates. Generative NVS models produce views that look plausible in isolation yet contradict each other geometrically. Adversarial or sloppy data collection introduces noise and repeated frames. The question a consistency metric must answer is not "can I produce geometry from these images" but rather "do these images provide evidence for a single coherent 3D explanation." Those are very different questions, and the paper "Can These Views Be One Scene? Evaluating Multiview 3D Consistency when 3D Foundation Models Hallucinate" (Paul, Kaushik, and Yuille, CCVL at Johns Hopkins) shows that conflating them produces a serious and under-examined failure mode.

The headline result is uncomfortable: modern 3D foundation models including VGGT, DUSt3R, MASt3R, and Fast3R will happily return dense geometry and cross-view support for inputs that correspond to no scene at all. Feed them three unrelated scenes, identical repeated frames, or pure Gaussian noise, and they hallucinate structure. Any metric built on top of these backbones, the canonical example being MEt3R, then inherits that hallucination and reports spuriously high consistency.

How the Failure Propagates Through MEt3R

To understand why this matters, it helps to look at how MEt3R works. It reconstructs view pairs using DUSt3R or MASt3R, warps dense DINOv2/FeatUp features from one view into another using the predicted geometry, and averages the pairwise feature disagreement across the set. Lower disagreement means higher reported consistency.

The paper identifies two distinct sources of fragility in this design, and the distinction is important because they fail differently:

• Aggregation fragility. Pairwise mean aggregation collapses the residual distribution into a single number. A mean is precisely the statistic least sensitive to outliers. If one view is from an unrelated scene, or if there is long-range geometric drift across the set, averaging the residuals dilutes that signal until it disappears.
• Backbone fragility. This is the deeper issue. The reconstruction model was trained to produce geometry. It is doing its job when it produces geometry. But a metric must decide whether the images support that geometry, and the backbone simply does not encode that decision. Once unsupported geometry is generated, no amount of clever residual aggregation can fully recover the correct judgment.

The authors formalize this with a clean abstraction: a parametric family that decomposes any neural evaluator into three components, a reconstruction backbone, a residual function, and an aggregation function. MEt3R is recovered as a single point in this family. This is a genuinely useful framing because it turns "is MEt3R good or bad" into "where in the design space does robustness come from."

What the Ablations Actually Show

The team builds SysCON3D, a controlled robustness benchmark. The corruptions are deliberately trivial: single outlier views, cross-scene mixtures, repeated images, patched noise, and full Gaussian noise. The premise is minimal and almost unarguable. An image set with contradictory or absent evidence for a shared scene should score worse than a clean multiview set. If a metric cannot pass this bar, it cannot be trusted on harder, realistic cases.

Two findings stand out from the diagnostic work:

1. Distributional aggregation helps, but only partially. Replacing mean aggregation with distribution-aware aggregation produces stronger metrics. Their best learned variant, MASt3R-W-IMQ, is reported as up to 3x more robust than MEt3R on SysCON3D. This confirms that the aggregation problem is real and partly fixable within the neural framework.

2. The backbone ceiling is real. Even with better aggregation, the learned metrics remain limited because the underlying reconstruction prior keeps fabricating support. You cannot post-process your way out of a backbone that confidently invents geometry for Gaussian noise.

The proposed alternative is to bring back classical geometric verification. The authors introduce COLMAP-based metrics that use feature matches, camera registration success, dense support, and crucially reconstruction failure itself as a failure-aware signal. The key philosophical move here is that classical structure-from-motion fails loudly. When COLMAP cannot register views, that failure is observable and informative, not hidden behind a smooth feature map. On real NVS outputs from Mip-NeRF360 and DL3DV, and in a structured human study designed to separate 3D consistency from mere visual realism, the COLMAP-based metrics achieve up to 4x higher correlation with human judgment than MEt3R.

My Take: Hallucination Is Not a Bug Here, It Is the Design

I want to push on the conceptual point because I think it generalizes well beyond this paper. There is an inherent tension in using a strong generative or reconstructive prior as an evaluator. The same inductive bias that makes VGGT and MASt3R good at completing plausible geometry from sparse, noisy real data is exactly what makes them poor at refusing to produce geometry when none is warranted. A prior strong enough to hallucinate is, by construction, a prior unwilling to say "I see nothing here."

This mirrors a pattern we have seen elsewhere. Language models fine-tuned for helpfulness become reluctant to refuse. Diffusion models produce confident detail in regions where the conditioning is uninformative. The failure mode is not laziness or under-training. It is the direct consequence of optimizing a model to always produce a coherent output. Evaluation, by contrast, requires the capacity to abstain, and abstention is not what these models were trained to do.

This is why the classical pipeline wins in the consistency setting. COLMAP has an explicit evidence test underneath: epipolar constraints, inlier counts, registration thresholds. It is not smarter than VGGT in any general sense; it is simply built around verification rather than generation. The paper's contribution, in my reading, is less about COLMAP being a good metric and more about demonstrating that failure-awareness must be designed in, not learned implicitly.

Limitations and the Honest Caveats

The authors are appropriately candid that COLMAP-based metrics inherit the well-known weaknesses of SfM and MVS. They are slow, they struggle under textureless surfaces, severe illumination changes, and very wide baselines, and they can fail on legitimate clean scenes that happen to be hard to match. The crucial difference they emphasize is that these failures are measurable rather than hidden, which is a reasonable claim, but it does shift the burden onto the practitioner to interpret failure correctly.

There is also a subtle risk worth flagging. A metric that rewards "COLMAP succeeds" could be gamed by NVS methods that generate easily-matchable texture even when the underlying geometry is wrong. Classical verification is harder to fool than feature warping, but it is not unfoolable. The human study results are reassuring on this point, but the adversarial robustness of these metrics against NVS systems trained against them remains open.

Open Questions

The tweet I posted earlier framed the central open question, and I still believe it is the right one: can any purely learned metric be failure-aware without an explicit evidence test underneath? The paper strongly suggests the answer is no, at least for current backbones. But that invites a constructive follow-up. Could we train a reconstruction model with an explicit "no scene" output, an abstention head supervised on negative examples like the SysCON3D corruptions? Such a model might combine the speed and flexibility of learned methods with the loud-failure property of classical verification.

For now, the practical lesson is concrete. If you are reporting 3D consistency for NVS or sparse-view reconstruction, do not trust a single learned scalar that was produced by a reconstruction prior. Pair it with a verification-based signal, inspect the residual distribution rather than its mean, and treat reconstruction failure as information rather than noise. The judge should not be the same machine that hallucinates the evidence.