A new method named ActCam allows for precise control over character motion and camera movement in generated videos, according to research published on arXiv. This approach operates as a "zero-shot" technique, meaning it can be applied to existing pre-trained image-to-video diffusion models without the need for further training. ActCam enables users to dictate both the actions of a character and the camera's trajectory within a generated scene.
The method works by transferring character motion from a source video into a new scene while simultaneously controlling intrinsic and extrinsic camera parameters on a per-frame basis. ActCam is designed to integrate with any pre-trained image-to-video diffusion model that can accept conditioning inputs such as scene depth and character pose. To achieve this joint control, ActCam generates pose and depth conditions that remain consistent across each frame of the video.
The generation process involves a two-phase conditioning schedule. In the initial stages of denoising, the model uses both pose and sparse depth information to establish the scene's structure. Subsequently, the depth conditioning is removed, and only the pose guidance is used to refine high-frequency details. This staged approach prevents over-constraining the generation process.
ActCam's developers evaluated the method across various benchmarks, testing it with diverse character motions and challenging viewpoint changes. Their findings indicate that ActCam improves camera adherence and motion fidelity when compared to methods that only control pose or other methods that attempt to control both pose and camera. Human evaluations showed a preference for ActCam, particularly in scenarios involving significant viewpoint shifts. The research highlights that careful camera-consistent conditioning and staged guidance are effective for achieving strong joint control over camera and motion without requiring model retraining.
The core idea behind ActCam is to provide the diffusion model with a target-view-aligned condition that jointly encodes both character and camera motion. The method utilizes depth as a geometric prior. When this depth information is rendered from the target camera's perspective, it implicitly conveys viewpoint changes through apparent background motion. A challenge addressed by ActCam is the potential conflict that can arise from using the reference image's depth, which might include a static character that clashes with the dynamic pose signal. ActCam resolves this by removing the character from the reference image and estimating a background-only depth map.
The researchers' contributions include the development of ActCam, a training-free method for joint acting-motion and camera-trajectory control in image-conditioned video generation. They also designed a novel conditioning pipeline that aligns motion (pose) and scene geometry (depth) to the target camera. This pipeline prevents static and dynamic interference through techniques such as reference character removal, geometry-aware placement, and depth alignment. The two-phase conditioning pipeline, as described earlier, is another key contribution.
The project page for ActCam is available at https://elkhomar.github.io/actcam/. Other recent work in controllable video generation includes RealCam, which focuses on real-time novel-view video generation with interactive camera control, and CameraCtrl, which enables camera pose control for video diffusion models by training a plug-and-play camera pose control module.