Evaluating the physical accuracy of artificial intelligence-generated videos has posed a significant challenge, often relying on subjective human judgment. A new quantitative framework, PDI-Bench (Perspective Distortion Index), has been developed to address this gap by auditing the geometric consistency of AI-created video content. The framework, detailed in a paper on arXiv, aims to provide a more objective measure of whether generated videos adhere to plausible 3D structure and motion principles.
Current methods for assessing video quality often fall short when it comes to geometric failures. Human evaluators, or even automated systems trained on human perception, can miss subtle but critical errors in how objects maintain their shape, scale, and movement in three-dimensional space. PDI-Bench seeks to overcome these limitations by transforming two-dimensional pixel data into verifiable three-dimensional geometric reasoning.
The PDI-Bench methodology operates through a multi-stage process. Initially, object-centric observations are extracted from a generated video clip using segmentation and point tracking tools, such as SAM 2, MegaSaM, and CoTracker3. These observations are then "lifted" into 3D world-space coordinates through monocular reconstruction. Finally, the framework computes a set of projective-geometry residuals to identify failures across three key dimensions: scale-depth alignment, 3D motion consistency, and 3D structural rigidity.
The "scale-depth alignment" component of PDI-Bench audits adherence to the pinhole camera model's invariant, which relates projected pixel height to 3D depth. Deviations here can indicate non-physical scaling. The "3D motion consistency" metric evaluates kinematic plausibility directly within the reconstructed world coordinate system, assessing velocity and acceleration. Lastly, "3D structural rigidity" quantifies non-physical internal deformations by examining the temporal consistency of pairwise distances between tracked anchor points in 3D space.
To support its evaluation capabilities, the researchers have also built PDI-Dataset. This dataset comprises real-world footage and synthetic videos generated by various AI models, specifically designed to test geometric constraints across scenarios like longitudinal convergence, dynamic tracking, biological motion, curved motion, and partial occlusion.
Initial tests of PDI-Bench on state-of-the-art video generators have revealed consistent geometry-specific failure modes that were previously undetected by common perceptual metrics. For instance, visually advanced models like Sora and HunyuanVideo reportedly exhibit severe "scale hallucinations," indicating a fundamental failure in maintaining perspective invariance. Real-world ground truth videos achieved a PDI score of 0.1206, while leading generative models like Seedance 2.0 and CogVideoX-3 scored 0.2422 and 0.2480, respectively.
The development of PDI-Bench is part of a broader effort within the research community to imbue generative video models with a more profound understanding of the physical world. Other recent work includes GeCo (Geometric Consistency), a metric that fuses motion and depth priors to detect deformation and occlusion inconsistencies in generated videos. GeCo can also be used as a training-free guidance loss to reduce geometric artifacts during generation. Additionally, metrics like the World Consistency Score (WCS) aim to provide a unified evaluation of internal world consistency, incorporating factors such as object permanence and relation stability.
The introduction of PDI-Bench and its associated dataset offers a valuable diagnostic tool for researchers and developers working on AI video generation. It provides a pathway toward creating models that not only produce visually appealing content but also adhere to the fundamental laws of physics, thereby advancing the goal of developing AI systems that can function as true implicit world models. The researchers have made their code and dataset publicly available.