Hugging Face integrates Nunchaku, enabling 4-bit diffusion model inference within its Diffusers library. This integration allows for reduced memory usage and faster image generation without requiring custom code.
The Nunchaku method, based on SVDQuant, quantizes both weights and activations to 4 bits. This approach significantly cuts down the memory footprint of large diffusion models, making them accessible on consumer hardware. Hugging Face's Diffusers library now supports Nunchaku directly, simplifying the process for users to load and run these optimized models.
Large diffusion models, capable of generating complex images and other media, typically demand substantial graphics processing unit (GPU) memory. Loading these models in standard BF16 precision can require 20 to 30 gigabytes (GB) of VRAM, a threshold beyond most consumer-grade GPUs. Quantization techniques aim to mitigate this by reducing the precision of model weights and activations. While existing methods in Diffusers, such as those using bitsandbytes or Quanto, reduce memory usage by storing weights in lower precision, they do not always accelerate inference speed.
Nunchaku takes a different path. Developed by Nunchaku AI, its SVDQuant quantization method operates on both weights and activations at 4-bit precision (W4A4). This dual quantization, combined with Nunchaku's specialized inference engine, not only slashes memory requirements but also speeds up the denoising process. According to Hugging Face's announcement, using Nunchaku Lite checkpoints can reduce peak VRAM usage by approximately 50% and offer a speedup of around 30% compared to standard pipelines.
The core innovation of SVDQuant lies in its handling of outliers, which are common in diffusion model weights and activations and pose challenges for aggressive quantization. SVDQuant addresses this by moving activation outliers into the weights and then representing the most challenging parts of each weight matrix with a small, 16-bit low-rank branch. The remaining residuals are quantized to 4 bits. The Nunchaku inference engine further optimizes this by fusing kernels for the 4-bit path and the low-rank branch, minimizing data movement overhead and enhancing speed.
Previously, utilizing Nunchaku-quantized models necessitated separate inference libraries. With the integration into Diffusers, loading a Nunchaku checkpoint is as straightforward as calling the `from_pretrained()` function. This eliminates the need for local CUDA compilation, thanks to an accompanying kernels package. The `diffuse-compressor` toolkit, also provided, allows users to quantize their own architectures and publish them as standard Diffusers repositories.
Nunchaku Lite checkpoints offer specific hardware requirements. For instance, NVFP4 checkpoints are designed for NVIDIA Blackwell GPUs (RTX 50 series and newer). For earlier GPU generations, INT4 variants are available. A Nunchaku Lite checkpoint paired with a bitsandbytes NF4 text encoder can generate a 1024x1024 image in approximately 1.7 seconds on an RTX 5090, consuming about 12 GB of VRAM, a substantial reduction from the roughly 24 GB needed for a BF16 pipeline.
Further optimizations are possible by combining Nunchaku Lite with other Diffusers features, such as model offloading to the CPU (`enable_model_cpu_offload()` and `enable_sequential_cpu_offload()`) for users with limited VRAM. Quantizing the text encoder, which can also consume significant memory, with methods like bitsandbytes NF4 can further decrease peak VRAM usage.
The Nunchaku research, detailed in the paper "SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models," has demonstrated significant memory reduction and speedup. For example, on a 12B FLUX.1-dev model, it achieved a 3.6x memory reduction compared to the BF16 model and an 8.7x speedup over the 16-bit model on a 16GB RTX 4090 GPU, outperforming an NF4 W4A16 baseline by 3x.
The integration of Nunchaku into Hugging Face's Diffusers library marks a significant step in making advanced diffusion models more accessible to a wider range of users and hardware.