The AEON framework generates explicit 4D Gaussian Splatting (4DGS) representations within the latent space of a pre-trained video diffusion model, enabling real-time rendering from various viewpoints and at different times. This approach aims to bridge the gap between video generative models, such as Sora, which excel at realistic appearance but lack 3D consistency, and 3D generative models, like Marble, which provide geometric consistency but often result in static outputs.
The core of AEON involves a generative model that operates within a text- or image-conditioned video latent space. This model produces renderable 4D graphics primitives in the form of 4D Gaussian Splatting. The researchers first trained a 4D reconstruction model to predict both 4DGS and camera poses from monocular videos, establishing an initial 4D decoder. They then developed a domain adapter to translate generative video latents into reconstructive 4D latents through a distillation process on the video variational autoencoder (VAE).
This unified framework allows AEON to support video-based world modeling by rendering 4DGS along predicted camera trajectories. It also constructs a persistent 4D environment that includes scene dynamics, facilitating realistic world simulation. The researchers stated that AEON supports a diverse range of world modeling applications and achieved state-of-the-art performance on standard benchmarks.
Previous efforts in generative video models, such as GEN3C, have attempted to incorporate 3D information to improve consistency and camera control. However, these models often relied on 3D caches or point clouds to guide generation, and while improving consistency, they still faced challenges in dynamic fluid motion or high-fidelity open-world physics. Other research has also highlighted the persistent issue of 3D spatial geometric inconsistencies in high-fidelity video generation, proposing metrics like SGC to evaluate these failures.
The development of 4D Gaussian Splatting itself has been a recent area of focus, with techniques emerging to generate dynamic 3D content from monocular video while maintaining temporal and spatial consistency. Models like SplatFlux have explored feed-forward 4D Gaussian reconstruction and generation from video latents, aiming for complete 4D scene construction without post-processing. AEON builds on these advancements by integrating the strengths of both video and 3D generative approaches within a unified framework, offering a single model that can produce both photorealistic and geometrically consistent dynamic scenes.
The Tsinghua University team, including Weiqi Zhang, Wenyuan Zhang, Yu-Shen Liu, and Junsheng Zhou, presented their work, which will be featured at NeurIPS 2026.