A new open pretraining recipe, Puro-2B, allows researchers and developers to train large language models (LLMs) on consumer-grade NVIDIA RTX 5090 GPUs, according to a report published on arXiv. This development addresses the substantial cost barrier that has limited access to LLM pretraining for many academic and open-source groups. The Puro-2B models, based on the Qwen2-1.5B architecture, were trained from scratch on up to 1.4 trillion tokens using FP8 precision.
Historically, pretraining LLMs has incurred prohibitive costs. For example, training a Llama-3.2-3B model can exceed $1.5 million, and reproducing SmolLM3-3B costs over $700,000. These figures place foundational model development largely out of reach for individual researchers or smaller institutions. The Puro-2B recipe aims to democratize this process by offering a hardware-accessible and cost-efficient method.
The RTX 5090 GPU, with its 32GB of GDDR7 memory and Blackwell Tensor Core architecture, offers a compelling balance of performance and consumer-friendly pricing. At an estimated cost of around $2,000, or rental rates as low as $0.76 per hour on platforms like Spheron, the RTX 5090 provides a more accessible alternative to enterprise-grade GPUs such as the H100, which can cost $3-$10 per hour. For models under 30 billion parameters, the RTX 5090 can deliver better cost-per-token economics for inference compared to the H100. Its 32GB VRAM allows for fine-tuning models up to 20-30 billion parameters and training smaller models from scratch.
The Puro-2B project utilizes FP8 precision, a technique that represents model parameters with fewer bits to improve GPU efficiency without sacrificing accuracy. This precision scaling is a key factor in reducing memory requirements and accelerating training times. NVIDIA's Blackwell GPUs, which include the RTX 5090, incorporate Microscaling (MX) formats that combine narrow floating-point data types with per-block scaling factors to enhance quantization. Researchers have found that specific MXFP8 data types, such as E4M3, help maintain pretraining accuracy.
The Qwen2 series of models, on which Puro-2B is based, also features architectural optimizations that reduce Key-Value (KV) cache size per token, lowering the memory footprint during long-context inference tasks. The Qwen2.5 series, an improved iteration, has scaled its high-quality pretraining datasets from 7 trillion to 18 trillion tokens, enhancing common sense, expert knowledge, and reasoning capabilities.
The development of cost-efficient pretraining methods aligns with broader efforts in the AI community to make LLM development more accessible. Other initiatives include frameworks like SPES, which supports sparse training on weakly connected nodes using distributed GPUs like NVIDIA L40S, reducing memory and communication costs. Similarly, models like Meta's Muse Glimmer are optimized to run on consumer-grade GPUs, with quantization techniques compressing 30-billion-parameter models to fit within 24GB or 32GB of VRAM.
The ability to pretrain LLMs on consumer hardware could stimulate innovation by allowing more researchers and developers to experiment with foundational model architectures. This shift could lead to a wider array of specialized models and training methodologies outside of large, well-funded organizations. The Puro-2B recipe, by providing an open and accessible pathway, contributes to this decentralization of LLM development.