QuantiPhyChallenge 2026
Registration deadline: October 9, 2026 · 23:59 AoENew teams and team edits close · Details

Team registration and edits to existing teams close on .

After this deadline, no new teams will be allowed, and edits to existing teams will not be permitted. Please register your team and finalize all team details before the deadline.

The final submission deadline remains .

Deadline updated: October 23, 2026 · 23:59 AoEDetails

Final submissions now close . Please submit your team’s final predictions by this deadline.

View the competition timeline.

Template updated · September 14, 202632 IDs corrected · Details & download

Dear QuantiPhy participants,

A participant reported that the wording of several questions in the official submission template stated a missing or dimensionally inconsistent output unit. We have verified the report, audited all 3,289 questions, and corrected the affected question text. No ground-truth values, IDs, or row order have changed. The affected rows are identified by the template’s id column (not row numbers).

Corrected target units (32 IDs)

CorrectionIDs
Speed — in meters → in m/s23, 1131, 1225, 1631, 1665, 1668
Acceleration — in m/s → in m/s21385, 1525, 2195, 2196, 2201, 2202, 2207, 2208
Acceleration — in cm/s → in cm/s23274
Displacement — in cm/s → in cm2668, 2673, 2678
Velocity — in m/s2 → in m/s3082, 3083
Distance — in m/s → in meters3204
Velocity — unit added: in cm/s2303, 2304, 2305, 2322, 2323, 2324, 2341, 2342, 2343
Distance — unit added: in meters1505
Velocity — unit added: in m/s3263

In addition, we normalized minor punctuation and wording (full-width question marks, double ??, a missing “in” before a unit, and one missing space). These changes do not alter the meaning of any question.

What you need to do

  1. Download the corrected template.
  2. If you have already prepared or submitted a CSV, no re-formatting is needed — the id column and row order are identical. Please only check that your parsed_value for the 32 IDs above is expressed in the corrected unit, and resubmit if you change anything.

We thank the participant who reported this, and apologize for the inconvenience.

— The QuantiPhy organizers,

Registration closed NeurIPS 2026 · Competition Track CVPR 2026 Project↗

How well can vision–language models do physics, with numerical accuracy?

The QuantiPhy Challenge — a NeurIPS 2026 Competition on Quantitative Physical Reasoning in Vision–Language Models.

GenRobot.AI
UST

QuantiPhy asks competitors to build VLMs that estimate sizes, velocities, and accelerations directly from video — across microscopic, macroscopic, and astronomical scales — and quantifies how faithfully their predictions are grounded in what the model actually sees.

Starter Kit Dataset
Registration closed
  1. 00Days
  2. 00Hrs
  3. 00Min
  4. 00Sec
Submission closes
  1. –Days
  2. –Hrs
  3. –Min
  4. –Sec

Workshop Day Agenda

| 13:00–15:15 | ICC Sydney

TimeSession
13:00–13:10Opening Remarks & Challenge Overview
13:10–13:30Winners presentations — Main Track
13:30–13:50Winners presentations — Open-Weight Track
13:55–14:55Invited Talks
Portrait of Ranjay Krishna Ranjay Krishna Portrait of Jiajun Wu Jiajun Wu
14:55–15:15Awards Ceremony & Closing Remarks
§ 01  Overview

A benchmark for physics, not plausibility.

Plausible-sounding answers are not the same as numerically correct ones. QuantiPhy probes whether a VLM is measuring or merely guessing.

Vision–language models are increasingly asked to reason about the physical world — estimating the speed of a falling object, the diameter of a cell, or the acceleration of a galaxy cluster. Yet recent results show that contemporary VLMs rely heavily on memorized world knowledge rather than on the actual video and text presented to them.

The QuantiPhy benchmark, introduced at CVPR 2026 and re-released here as a NeurIPS competition, exposes this gap. Across 3,289 video–question pairs drawn from simulation, laboratory, and in-the-wild sources, the top frontier model trails the human baseline by more than two points — and collapses entirely under counterfactual prompts.

The QuantiPhy Challenge invites the community to close that gap, not only on raw accuracy but on the harder question that this benchmark was built to ask: are model predictions faithful to what the model actually sees?

  1. O.01

    Improve numerical accuracy on quantitative physics tasks

    Push Mean Relative Accuracy on size, velocity, and acceleration estimation toward the human ceiling.

  2. O.02

    Reward open, reproducible solutions

    A dedicated Open-Weight Track ensures the strongest methods remain reproducible and accessible to the broader community.

  3. O.03

    Generalize across scale

    Solutions must hold from microscopic to astronomical regimes — the same physical relationships, observed at vastly different magnitudes.

§ 02  Tracks

Two tracks, one benchmark.

Competitors may enter either or both tracks. Each track has its own leaderboard, evaluation protocol, and awards.

Track A MRA · Open

Main Track

The headline competition. Any model is permitted — proprietary, open-weight, or hybrid — and the only objective is raw numerical accuracy on the original QuantiPhy test set.

  • MetricMean Relative Accuracy
  • Models allowedAny
  • Code disclosureTop-3 only
Track B MRA · Open-weight

Open-Weight Track

Same scoring rule as the Main Track, but submissions must be based on publicly available model weights and tools — supporting reproducible, community-accessible research on quantitative physical reasoning.

  • MetricMean Relative Accuracy
  • Models allowedPublicly available weights
  • Code disclosureAll entries
§ 03  Dataset

3,289 questions. Three observation regimes. Three scales.

QuantiPhy is grounded in a multi-source video corpus designed to isolate what a model knows from what a model sees.

  • 3,289★ Video–question pairs
  • 568 Unique source videos
  • 3 Observation regimes
  • 10±25 Decades of physical scale

Simulation (Blender)

Fully controlled scenes with exact ground-truth annotations for position, velocity, and acceleration — the cleanest signal for diagnostic evaluation.

Laboratory

Multi-camera, calibrated recordings of real-world physical setups — controlled enough for ground truth, real enough to expose perception failure modes.

Internet (In-the-Wild)

Curated real-world videos with expert annotations, including footage from natural phenomena, sports, microscopy, and astronomical observation.

2D · planar
3D · depth
Static
static prior
2S static · 2D
3S static · 3D
Dynamic
dynamic prior
2D dynamic · 2D
3D dynamic · 3D

Four diagnostic categories (2S, 2D, 3S, 3D). Across all of them, the target quantities are size, velocity, and acceleration. Final scores are macro-averaged over the four categories.

★ Dataset, validation split, and a CC-licensed starter sample available on Hugging Face.

§ 04  Evaluation

Mean Relative Accuracy.

Both tracks share the original QuantiPhy metric — a threshold-averaged measure that rewards approximate correctness rather than exact matches.

A prediction ŷ for ground truth y is judged correct at confidence threshold θ when:

∣ŷ − y∣ ∣y∣ < 1 − θ

MRA averages the indicator over a set of D instances and a set of thresholds C = {0.10, 0.20, …, 0.90, 0.95}:

MRA(D) = 1 ∣D∣·∣C∣ ∑ i∈D ∑ θ∈C 𝟙( ∣ŷi − yi∣ ∣yi∣ < 1 − θ)

The final track score is the macro-average of MRA across the four QuantiPhy categories — c ∈ {2S, 2D, 3S, 3D} — giving equal weight to each prior×dimensionality setting:

Score = 1 4 ∑ c MRA(Dc)
§ 05  Timeline

Important dates.

All deadlines are 23:59 anywhere-on-Earth (AOE). Subject to ratification by the NeurIPS 2026 competition chairs.

  1. June 15–30, 2026
    Website, platform, and dry run

    Launch the official challenge website, publish rules and documentation, release the validation set with ground-truth answers, test the CSV evaluator, and conduct an internal dry run of the real-time leaderboard.

  2. Late June 2026
    Official launch

    Open registration and public submissions. Release the starting kit, baseline scripts, example CSV files, evaluation code for the validation set, and documentation.

  3. July–September 2026
    Development phase

    Participants submit CSV predictions to the online leaderboard and receive real-time scores. Organizers maintain the website, answer questions, update FAQs, and monitor submission integrity.

  4. Registration deadline

    Register your team and finalize all team details by 23:59 anywhere-on-Earth (AoE). After this deadline, no new teams will be allowed, and edits to existing teams will not be permitted.

  5. Final submission deadline

    Final CSV predictions must be submitted by 23:59 anywhere-on-Earth (AoE). Top teams are then asked to provide code or Docker containers for reproducibility verification.

  6. Late October 2026
    Verification and final ranking

    Organizers verify top submissions, run reproducibility checks when applicable, compute final scores and bootstrap confidence intervals, and finalize rankings.

  7. November 2026
    Result analysis and workshop preparation

    Prepare the competition report, invite top teams to submit short technical summaries, finalize workshop talks/posters, and publish final leaderboard analysis.

  8. December 11–12, 2026
    NeurIPS Competition Track workshop

    Present challenge design, final results, analysis of model performance, and selected participant methods at the in-person NeurIPS Competition Track workshop.

§ 06  Participation

How to compete.

Steps

  1. Register your team (registration closed; up to five members).
  2. Review the QuantiPhy dataset and the starter kit.
  3. Prepare a single .csv file matching the reference submission template.
  4. Sign in and upload your .csv in the submission portal — scored against the hidden test set on upload (up to 3 scored submissions per day).

Rules

  • Teams of up to five. One team per person, per track.
  • Submissions are made through the official evaluation server. The test split is hidden.
  • Track B (Open-Weight): submissions must be based on publicly available model weights and tools.
  • Top-3 entries publish a reproducibility package within 14 days of notification.
  • Organizers and direct collaborators are not eligible for prizes.
§ 07  Awards

Prizes

1st · per track $1,000

Track winners will be invited to a 10-minute oral presentation at the live workshop.

2nd · per track $500

Track winners will be invited to a 5-minute oral presentation at the live workshop.

3rd · per track $250

Track winners will be invited to a 5-minute oral presentation at the live workshop.

§ 08  Organizers

The QuantiPhy Team

Core organizers

Faculty advisors

§ 09  FAQ

Frequently asked.

Don't see your question? Email quantiphybench@gmail.com.

Can I enter both tracks?

Yes. The two tracks share a registration but maintain independent leaderboards. A single submission file template covers both; you opt in to each track per upload.

Does the Open-Weight Track allow proprietary APIs as part of a pipeline?

No. Every model invoked at inference time must be based on publicly available weights and tools at the time of submission.

Will the test ground truth be released after the competition?

No. The numerical ground truth for the test set remains withheld so QuantiPhy can serve as a reliable long-term benchmark beyond NeurIPS. A small validation set with ground-truth labels is provided for prompt development, local evaluation, and sanity checks.

What format are submissions in?

A CSV file containing the numerical prediction for each test instance, in the required output unit. The evaluation server scores submissions against the protected ground truth automatically.

Can I use Main Track submissions to bootstrap an Open-Weight Track submission?

You may, provided the Open-Weight pipeline at inference time uses only publicly available weights and tools. Closed-API outputs used during method development must be disclosed in the technical report.

§ 11  Citation

If you use QuantiPhy, please cite.

@article{li2025quantiphy,
  title   = {QuantiPhy: A Quantitative Benchmark Evaluating Physical
             Reasoning Abilities of Vision-Language Models},
  author  = {Li, Puyin and Xiang, Tiange and Mao, Ella and Wei, Shirley
             and Chen, Xinye and Masood, Adnan and Li, Fei-Fei and Adeli, Ehsan},
  journal = {arXiv preprint arXiv:2512.19526},
  year    = 2025
}