LAION-BVD Dataset Offers 10 Million Hours of Video for Open Multimodal Research

LAION-BVD Dataset Offers 10 Million Hours of Video for Open Multimodal Research

A new open dataset named LAION-BVD has been released, providing 10 million hours of video content for multimodal pre-training. The dataset, compiled from 80 million downloaded videos, aims to increase accessibility for researchers in video, audio, and image modalities.

The LAION-BVD dataset, detailed in a paper on arXiv, comprises 1.3 billion video URLs sourced from Common Crawl. Researchers downloaded 80 million videos from these URLs, accumulating a total duration of 10 million hours. The dataset is designed for multimodal pre-training, enabling models to learn across video, audio, and image data. To facilitate this, the creators used content-aware scene detection to extract clips and synthetically generated captions for both video and audio. Additionally, the dataset includes scene-changing frames extracted from videos, which offer a distinct visual distribution compared to typical web image corpora and can be used for image-text tasks.

Models trained on LAION-BVD have demonstrated competitive performance on standard video-text and audio-text benchmarks. The paper reports consistent improvements as the training or model scale increases. For instance, ViCLIP models trained on LAION-BVD matched or surpassed InternVid-trained models by up to 4.4% on video-text benchmarks. CLAP models trained on the dataset also achieved competitive results against other large-scale audio datasets, utilizing the in-the-wild soundscapes from the videos. Frame-based CLIP models trained on the extracted frames showed strong performance in image-text retrieval.

The LAION-BVD dataset is released under a CC-BY-4.0 license, permitting commercial use with attribution. This licensing, combined with its scale, distinguishes it from previous open video efforts. The creators stated that the dataset is intended to support open and reproducible multimodal research. They noted that large-scale video datasets and the models trained on them are increasingly concentrated within a few technology companies, limiting independent scientific investigation. By providing this open resource, the goal is to broaden access to multimodal training data and allow for more transparent evaluation of large-scale models.

LAION, the organization behind the dataset, is a non-profit entity known for releasing open-sourced AI models and datasets. Previously, LAION released large image-caption datasets like LAION-400M and LAION-5B, which have been used to train prominent text-to-image models such as Stable Diffusion. The organization has faced legal challenges regarding its datasets, including lawsuits from Getty Images and a German photographer, though a German court dismissed one such lawsuit in September 2024. LAION also previously released a cleaned dataset called Re-LAION-5B in August 2024 after investigations found problematic content in earlier versions.

The development pipeline for LAION-BVD involved several stages: bulk video acquisition from Common Crawl URLs, curation including scene detection and splitting, frame extraction, and captioning using Vision-Language Models (VLMs). The dataset also includes 55 million annotated clips and 300 million extracted frames. The release of LAION-BVD is intended to support scientific research, reproducibility, safety analysis, and the study of multimodal foundation models.