QuantiPhy introduces a rigorous benchmark for evaluating quantitative physical reasoning in Vision-Language Models. Unlike traditional VQA tasks that focus on qualitative descriptions, QuantiPhy challenges models to perform precise numerical inference grounded in physical laws.

• 🆕 Novel Task: Kinematic Inference

  We formally define a task where object size, velocity, and acceleration are treated as mutually constraining quantities.

  • Input: A video clip + A single physical prior provided as text (e.g., "The car is 4 meters long" or "Gravity is 9.8 m/s²").
  • Reasoning: The model must use the provided prior to recover the world-to-pixel scale and leverage kinematic equations to deduce other unknown properties of the target object.
  • Output: A precise numerical value (with units) for a target property (e.g., "The velocity at t=2s is 12.5 m/s").
• 🗂️ Structured Taxonomy

  To provide fine-grained analysis of model capabilities, the benchmark is organized along two primary axes:

  • Dimensionality: 2D (Planar motion) vs. 3D (Depth-varying motion).
  • Physical Prior: Static (Size-based) vs. Dynamic (Motion-based, e.g., Velocity/Acceleration).
• 📊 Scale & Diversity

  The dataset contains 3,355 video-question pairs derived from 569 unique videos. The data spans diverse sources (Simulation, Lab, Internet) to ensure coverage across microscopic, macroscopic, and astronomical scales.

QuantiPhy is built from four complementary data sources, each designed to probe quantitative physical reasoning under different visual conditions. For every video, we segment object trajectories and annotate size, velocity, and acceleration using source-appropriate, physically grounded procedures.

• 🔧 Simulation (Fully Controlled)
  • Collection: Physically grounded Blender simulations with explicitly controlled object properties and motions
  • Segmentation: Automatic object-level segmentation from the simulation engine
  • Annotation: Exact ground-truth physical quantities directly obtained from simulation parameters
  • Key advantage: Enables precise causal and counterfactual interventions.
• 🧪 Laboratory (Measured Reality)
  • Collection: Multi-camera recordings of real objects in motion under calibrated setups
  • Segmentation: Frame-level object segmentation via semi-automatic pipelines with manual verification
  • Annotation: Physical quantities computed from measured trajectories and known object dimensions
  • Key advantage: Provides real-world grounding with reliable physical measurements.
• 🌍 Internet (In-the-Wild)
  • Collection: Curated real-world videos with visible, trackable object motion
  • Segmentation: Manual and model-assisted object segmentation under diverse visual conditions
  • Annotation: Expert annotations constrained by visual evidence and physical plausibility
  • Key advantage: Tests robustness under natural noise, clutter, and ambiguity.

We first examine how scene context affects quantitative reasoning by varying background complexity and the number of moving objects. Surprisingly, background complexity alone has only a mild effect on performance. While removing clutter via segmentation slightly stabilizes predictions, models often perform equally well—or even better—in visually rich scenes, likely due to the presence of implicit reference cues such as tiles, road markings, or architectural structures.

In contrast, the number of objects in a scene has a consistent and substantial impact. Across models, multi-object scenes outperform single-object setups, suggesting that relational structure provides valuable comparative anchors for inferring both size and motion.

Figure 5. Performance across background complexity and object count conditions.

"Do VLMs actually use what we give them?"

We next investigate whether VLMs faithfully condition on the reference video and the explicit physical prior provided in the prompt. We evaluate models under three controlled settings: the default video + prior condition, a prior-only condition with the video removed, and a counterfactual prior condition where the numerical prior is systematically rescaled.

If models relied on visual measurement and algebraic reasoning, removing the video or altering the prior should lead to substantial performance degradation. Instead, we observe a striking pattern: many models achieve comparable accuracy even without the video, indicating that plausible guesses based on memorized world knowledge already suffice. When the prior is counterfactually rescaled, performance collapses across nearly all models, with predictions remaining anchored to typical real-world magnitudes rather than the provided numerical values.

Finally, we test whether structured chain-of-thought (CoT) prompting can mitigate these failures. Contrary to expectation, step-by-step decomposition does not systematically improve performance. For most models, explicitly reasoning through pixel measurement and scale estimation amplifies early numerical errors rather than correcting them.

Table 2. Input faithfulness analysis across different experimental conditions.