Introducing CaliBench: Are World Models Physically Calibrated?
CaliBench tests whether video world models reproduce the true distribution of physical outcomes—not just whether each frame appears physically accurate
Jon Sadeghi
August 10th, 2026
Odyssey is building foundation world models: causal, multimodal systems that learn how the world evolves over time. For a world model to stand in for reality—in robotics, science, healthcare, education, or gaming—it has to do more than render convincing video. The world model must accurately reproduce the characteristics of the world which are relevant to its deployment setting. Crucially, these characteristics may not be aligned with human aesthetic preferences alone.
What Makes a Perfect Simulation?
In The Matrix, the simulation gives itself away not through a dramatic failure but through a small repetition—the same black cat walking past twice, a déjà vu too improbable to be chance. A convincing simulation can betray itself through the statistics of its outcomes, not just through broken visuals. World models face the same test: what looks like randomness in a video—a dice roll, a coin flip—is really a sensitive dependence on initial conditions that appears random. To faithfully reproduce reality the model must be random, i.e. sample a distribution of outcomes rather than a single trajectory. The question is whether that distribution is right.
Drop a ball through a Galton board, roll a die, or spin a roulette wheel and the outcome is genuinely uncertain—but the distribution of outcomes is not. A ball is most likely to land in the centre bin and rarely at the edges. A fair die lands on each face one time in six. A faithful simulator should reproduce those distributions—not just one plausible-looking result. This is referred to as calibration, and it turns out today's best video models are not very good at it.
Per-Video Realism Is Not the Same as Calibration
Most video benchmarks measure one of two things. Either they judge each individual generation—is it sharp, temporally consistent, physically plausible frame by frame—or they compare entire datasets in an abstract learned feature space, as Fréchet Video Distance (FVD) does. Both are useful, and neither answers our question: across many generations from the same starting state, does the model reproduce the correct distribution of physical outcomes?
A model can pass every per-sample check—every video looks great—and still be badly miscalibrated, collapsing onto the single most likely outcome every time. CaliBench is built to measure exactly that.
Stochastic Dynamics with Interpretable Outcomes
The core idea is to choose physical systems where an outcome distribution can be extracted and compared to some reference distribution. For the first release of CaliBench, we choose systems where the outcome distribution is interpretable, known exactly in closed form, and has discrete support:
The Galton board
A binomial distribution over which bin the ball lands in.
A ball rolling down a Y-fork, a person reaching a T-junction, a swinging double pendulum
A fair coin flip, left or right.
A dice, a drawn card, a lottery tumbler
Uniform over faces, suits, or numbers.
A European roulette wheel
A known, skewed distribution over red, black, and the rare green.
Because we know the true distribution analytically, we can measure exactly how far a model's outputs sit from it—in a physically meaningful, and human-interpretable feature space. The outcome of each video is a bin index, a die face, a suit, or a colour, which we read automatically with a vision-language model (Gemini 3.1 Pro) and validate against human annotators at around 93% agreement.
Two Ways a Simulation Can Fail
We deliberately separate two failures that a single accuracy number would blur together:
Scorability—the fraction of generations that produce a clean, readable outcome at all. A ball that settles in one bin is scorable; a video that spawns three balls, or a pendulum that tears itself apart, is not.
Calibration—among the readable outcomes, how far the distribution sits from the true reference, measured by total variation distance.
These are different failures. A model can produce few valid videos, but distribute them accurately, or produce flawless-looking videos that all land on the same outcome. We also run a statistical test to check whether an observed gap is real or just sampling noise, and report a single summary metric—mean normalised total variation, or mnTV—to enable a single ranking of models to be constructed across all experiments.
Realistic Frames, Unrealistically Distributed
We ran six frontier image-to-video models—WAN-2.7, SeeDance-2.0, HappyHorse-1.0, Veo 3.1, Runway Gen-4.5, and Cosmos3-Super—across all nine scenes with 32 generations for each model.
The dominant failure is mode collapse. Rather than reproducing the spread of outcomes, models pile probability mass onto a small handful of them. Most scene–model pairs are significantly miscalibrated, and even the best aggregate score sits far above where a perfectly calibrated model would land. On the Galton boards, most models over-concentrate on the central bin well past its true 24.6% peak. And in the most extreme case, Veo 3.1 demonstrates total mode collapse, outputting the exact same die face across every random seed.
Strikingly, there is no overall winner. Performance varies sharply by scene, not by model. For example, whilst SeeDance performs best overall, HappyHorse performs better on some individual scenes. We did not find any simple law enabling us to predict calibration performance based on a particular characteristic of the model or scene.
We did find that classifier-free guidance behaves like a dial. Turning it down reduces over-concentration and increases predictive entropy, but it trades away structural scorability at the same time—a tension between rendering valid frames and reproducing the right distribution.
Why Calibration Is a World Model Problem
If we want to use world models to stress-test autonomous systems—self-driving stacks, robot policies—in simulation, then reproducing the full variance of plausible futures is not optional. A simulator that collapses to the single likeliest outcome will hide precisely the rare, dangerous tail events you built the simulation to surface. You would come away reassured by a world that is far more deterministic, and safer, than the real one.
This is also how we think about world models at Odyssey. To answer the question we posed: a perfect simulator is not one that produces a single beautiful future—it is one that gets the distribution over many possible futures right. Calibration is part of what separates a model that looks like a simulator from one that is one.
What We're Releasing Today
Today, we release the full CaliBench protocol—the curated conditioning frames, the standardised VLM extraction prompts, and the analysis code—along with the mnTV metric, so new models can be compared directly against the results here.
What's Next?
The field has spent a decade making each frame of a simulation look real. Making the distribution of outcomes real is a different challenge, and an open one. As the numbers above show, today's best models are nowhere near solving it. We're releasing CaliBench so the community can measure that gap and close it together: run the protocol, report your mnTV next to your FVD and VBench scores, and treat driving it toward zero as a first-class goal. We think calibration, just like realism, should be a core target for the next generation of world models.
The open problems are immense—how to build distributional fidelity into pre and post-training, and how to close the gap without trading away per-video realism. If you're working on any of them, we'd love to compare notes—and to build a calibrated world simulator with you.
The Team That Brought This to Life
CaliBench was made possible by the incredible Odyssey team—Jonathan Sadeghi, Jenny Seidenschwarz, Jesse Allardice, Sirish Srinivasan, Benjamin Graham, and Jeffrey Hawke.
