A new benchmark named EntityBench has been developed to evaluate the ability of artificial intelligence models to maintain consistency in characters, objects, and locations across multiple generated video shots. This development, detailed in a paper published on arXiv, addresses a growing challenge in AI video generation as systems move from creating single, static scenes to generating longer, narrative visual sequences.
Current methods for assessing AI video generation often rely on sets of prompts that are generated independently, limiting the scope of entity coverage and employing simple consistency metrics. This makes standardized comparisons between different AI models difficult and hinders the systematic diagnosis of where and why entity consistency breaks down over time. EntityBench aims to rectify this by providing a structured dataset and a multi-faceted evaluation framework.
The EntityBench benchmark comprises 140 distinct episodes, totaling 2,491 individual shots. These episodes are derived from real narrative media and are organized into three difficulty tiers: easy, medium, and hard. The benchmark supports sequences of up to 50 shots, tracking up to 13 characters, 8 locations, and 22 objects simultaneously across these shots. A key feature is the explicit entity schedule for each shot, detailing which specific entities should appear. This allows for the assessment of recurrence gaps, which can span up to 48 shots, providing a rigorous test of an AI's memory and consistency capabilities.
To accompany the dataset, the researchers have designed a three-pillar evaluation framework. The first pillar assesses intra-shot visual quality, examining the realism and coherence of individual video frames. The second pillar measures prompt-following alignment, verifying if the generated video accurately reflects the textual descriptions provided. The third and most critical pillar focuses on cross-shot entity consistency. This pillar incorporates a fidelity gate, ensuring that only accurately rendered entities are considered when scoring consistency. This prevents errors in initial entity generation from unfairly penalizing later consistency assessments.
As part of their research, the team also proposed a baseline system called EntityMem. This system uses a memory-augmented approach to enhance entity consistency. Before video generation begins, EntityMem plans and stores verified visual references for each entity in a persistent memory bank. The video generation backbone can then retrieve these references to maintain visual continuity across shots. Experiments conducted using EntityBench indicate that existing methods show a sharp decline in cross-shot entity consistency as the recurrence distance between shots increases. EntityMem, however, demonstrated superior character fidelity, with a Cohen's d value of +2.33, suggesting a substantial practical improvement.
The development of EntityBench marks a significant step towards more standardized and rigorous evaluation in the field of multi-shot video generation. By providing a comprehensive dataset and a detailed evaluation framework, it offers researchers a clearer path to understanding and improving the ability of AI models to generate coherent and consistent visual narratives over longer sequences.