A new research paper details S$^3$T (Self-Supervised Self-Distillation over Time), a framework designed to track visual states in videos without requiring human annotations or external supervision. The system, developed by Shravan Venkatraman, Wenshuai Zhao, Mohammad Hassan Vali, and Arno Solin, uses a self-contained approach where the model generates its own training targets. This eliminates the need for labeled data, separate teacher models, or reward signals, and does not add to inference costs.
S$^3$T operates on the principle that a denser temporal sampling of a video clip provides a more accurate representation of the running visual state. This denser view acts as a "teacher" for a "student" model that processes a sparser view of the same clip. Both teacher and student share the same weights, with the student learning to match the teacher's next-token distribution. The training process involves sampling a full video clip twice, with the student reading 12 uniformly spaced frames and the teacher reading 24 frames, both covering the same temporal extent. The sparse student then generates an answer to a fixed state question, and the dense teacher provides the positive target for updating the student adapter.
The researchers evaluated S$^3$T's performance on the LLaVA-OneVision-2-8B benchmark. As a single model, S$^3$T improved VSTAT accuracy by 1.74 points. When combining multiple models through a technique called "souping," the accuracy gain increased to 2.38 points. Further adaptation of the vision encoder led to a 2.70-point improvement in VSTAT accuracy. These results indicate that S$^3$T significantly enhances state tracking capabilities compared to prior self-evolving methods, which showed little change in this area.
The training for S$^3$T uses only unlabeled synthetic video clips. Despite this, the learned capabilities transfer effectively to real-world videos. On VSTAT-YouTube state-tracking questions, S$^3$T demonstrated an improvement of 7.95 points. The framework also improved performance on the MVBench Action Count by 4.50 points. This transferability from synthetic to real data highlights the method's potential for practical applications.
The core idea of treating temporal sampling density as privileged information for self-distillation differentiates S$^3$T from other self-supervised video representation learning approaches. Some prior work in self-supervised video tasks has focused on masked video modeling or temporal correspondence for representation learning. For instance, the T-CoRe framework uses a sandwich sampling strategy to reduce reconstruction uncertainty and an auxiliary branch in a self-distillation architecture to restore representations in the latent space. Other methods have explored distillation learning for video object segmentation, often employing complex architectures that can be computationally intensive. S$^3$T's approach aims for a self-contained framework that avoids such complexities and external dependencies.
The researchers involved in the S$^3$T project are affiliated with the Mohamed bin Zayed University of Artificial Intelligence, ELLIS Institute Finland, and Aalto University. Their work contributes to the broader field of self-supervised learning for video analysis, an area that has seen increasing research interest in recent years due to its ability to learn from large amounts of unlabeled data.