A new training method named Sonar-MoE reduces the peak GPU memory required for Mixture-of-Experts (MoE) models by updating expert tensors during the backward pass. This technique lowers memory consumption by 19% to 22% when fine-tuning models such as OLMoE-1B-7B and Granite-3.1-3B-A800M in bf16 precision, according to a paper published on arXiv. The core innovation lies in adapting an "optimizer-in-backward" approach, previously used for dense models, to the batched expert tensors in MoE architectures.Traditional MoE training retains all expert gradients in GPU memory until the optimizer step. As MoE models scale with more and larger experts, this memory requirement increases, creating a bottleneck. Sonar-MoE addresses this by updating each expert tensor and then immediately freeing its gradient, ensuring the GPU only stores one layer of expert gradients at a time. The method produces updates that are bitwise identical to standard training, provided no expert gradient clipping is used, and maintains held-out loss within 0.003 of baseline methods.The development of memory-efficient training methods is critical as MoE models become more prevalent in large-scale machine learning. MoE architectures enhance computational efficiency by activating only a subset of "experts" (model weights) for each input, thereby increasing model capacity without a proportional increase in computational cost. However, their large parameter count often leads to substantial memory overheads, particularly during training.Other research efforts also aim to mitigate the memory challenges in MoE training. For instance, SonicMoE, developed by researchers from Princeton University, UC Berkeley, and Together AI, focuses on hardware and model architecture co-design to improve MoE efficiency. SonicMoE reduces activation memory by up to 45% and achieves a 1.86x improvement in compute throughput on Hopper GPUs by using a "token rounding" routing method and GPU kernels that overlap memory I/O with computation. This allows for similar work with fewer GPUs, cutting hardware needs by approximately 33%.Another approach, RelayMoE, proposes a ring-based execution model to avoid large intermediate buffers during distributed MoE training. It reconstructs expert intermediates, computes gradients, and releases them in a hop-by-hop fashion, supporting longer sequences and larger batches. RelayMoE has shown up to a 2.02x throughput improvement and extended trainable sequence length by up to 2.85x in full-model training within the same GPU memory budget.MoEBlaze, a framework presented at MLSys 2026, also tackles memory efficiency through optimized data structures and co-designed kernels with activation checkpointing. MoEBlaze claims over 4x speedups and more than 50% memory savings compared to existing MoE frameworks. Additionally, the Mixture-of-Parallelisms (MoP) training stack specializes different parallelism techniques to various components of the MoE block, aiming for maximal efficiency by addressing component-specific bottlenecks.These advancements, including Sonar-MoE, highlight a broader trend in machine learning research to optimize the training of large-scale models, particularly MoE architectures, by addressing their significant memory demands. The goal is to enable the development and deployment of increasingly complex models with existing or even reduced hardware resources.