A recent paper published on arXiv, "Spectral and Subspace Dynamics of LoRA in Supervised Fine-Tuning and Reinforcement Learning," details how Low-Rank Adaptation (LoRA) exhibits varied learning dynamics when applied to supervised fine-tuning (SFT) compared to reinforcement learning (RL). The authors, Chenruo Liu, Zihan Wang, Yijun Dong, and Qi Lei from New York University and the Institute of Science and Technology Austria, found that a technique called group-mean baseline subtraction drives these observed differences.
LoRA is a parameter-efficient fine-tuning method that reduces computational costs by introducing small, trainable matrices alongside the original pre-trained weights. This approach allows for efficient adaptation of large language models (LLMs) to specific tasks. While LoRA is widely used in both SFT and RL, the mechanisms underlying its effectiveness in each stage had remained less clear.
The researchers tracked the singular spectrum and singular subspaces of LoRA updates during mathematical reasoning tasks. Their analysis showed that in RL, the singular spectrum tends to concentrate at all learning rates. In contrast, SFT expands the spectrum when moderate learning rates are used. Furthermore, the study observed that RL causes the singular subspaces to rotate at a much faster rate than SFT, particularly at smaller learning rates.
To pinpoint the source of these contrasting dynamics, the research team systematically introduced individual components, interpolating between SFT and RL setups. This process led them to identify group-mean baseline subtraction as the key factor. This technique, often employed in RL, involves subtracting the average outcome of a group from individual outcomes, which can influence how gradients are calculated and applied.
The paper further explains the underlying mechanism through a quadratic multi-index model updated via LoRA. Group-mean baseline subtraction, according to the authors, shifts the balance between two competing forces: coherent fitting and gradient noise. This shift promotes spectral concentration and faster subspace rotation in RL, while also constraining the risk-minimizing learning rate to a smaller value.
These findings suggest that LoRA updates behave differently depending on the learning paradigm. Previous research has explored LoRA's performance in both SFT and RL, noting that LoRA can achieve results comparable to full fine-tuning, especially on smaller datasets for SFT and even with low ranks for RL. Other studies have investigated LoRA's learning dynamics from a gradient flow perspective and explored methods to optimize its initialization and performance. The current study by Liu et al. provides a deeper understanding of the internal mechanics that differentiate LoRA's behavior in these two distinct learning environments.
The authors suggest that these insights could lead to tailored LoRA update strategies for each phase. For SFT, they propose smoothing the spectrum of the entire update to improve adaptation. This work contributes to a more nuanced understanding of how LoRA operates, potentially enabling more effective and efficient fine-tuning of models in diverse applications.