A new technique called SpectrumQuant has been developed to improve the efficiency of large language models (LLMs) through quantization. This method uses spectral alignment derived from activation covariance to enhance post-training quantization. Researchers propose that by maximizing covariance, they can achieve spectral alignment, which aids in constructing orthogonal transforms for quantization-friendly LLMs.
SpectrumQuant introduces a closed-form rotation method that estimates activation covariance from calibration data. This estimation is then used to construct a covariance-aware orthogonal transform. The researchers suggest this approach can improve quantization robustness. Experiments on LLaMA-3 8B demonstrated that SpectrumQuant can achieve 4-bit quantization for both weights and activations, reducing the accuracy gap to 1.5% compared to full precision. This method also provides a 2x faster inference speed and a 3x reduction in memory usage.
The paper highlights that existing quantization methods often rely on fixed data-agnostic transforms or calibration-time optimization of learned rotations. SpectrumQuant's approach is rooted in studying rotation selection from first principles. By focusing on maximizing covariance, the method aims to align the spectral properties of activations, making them more amenable to low-bit quantization. This is particularly important as LLMs continue to grow in size and complexity, requiring more efficient deployment strategies on consumer hardware.
The technique involves two main stages. First, activation outliers are smoothed and transferred into the weight matrix. This step simplifies the subsequent quantization process. Second, the method applies channel-wise low-frequency Fourier truncation. This process suppresses high-frequency components while retaining essential signal energy, thereby enhancing quantization robustness. The underlying principle is that most of the weight energy is concentrated in low-frequency components, which can be preserved with minimal impact on model accuracy.
For runtime adaptability, SpectrumQuant incorporates a lightweight truncation module during inference. This module adjusts truncation thresholds based on channel characteristics, allowing for dynamic optimization. The researchers have made the code for SpectrumQuant available.
This development comes amid a broader push for advanced quantization techniques to enable efficient LLM deployment on end-user devices. Other methods in this field include SmoothQuant, OPTQ, QuIP, SpinQuant, QuaRot, SVDQuant, and QuIP#. These techniques aim to mitigate activation and weight outliers, which can expand the quantization dynamic range and lead to significant accuracy degradation. However, challenges remain, with migration-based methods struggling with extreme outlier distributions, rotation-based techniques introducing inference overhead, and global decomposition methods failing to capture channel-specific patterns. SpectrumQuant's spectral alignment approach offers a principled way to address these limitations.