Researchers at Shodh AI have developed a new architecture named Shodh-MoE to address a significant challenge in scaling scientific machine learning models. The problem, known as negative transfer, occurs when training a single dense neural network on diverse physical systems. This can lead to conflicting gradients, unstable optimization, and a loss of model adaptability. The proposed solution, detailed in a paper on arXiv, employs a sparse mixture-of-experts approach to route computations, allowing different parts of the model to specialize in distinct physical phenomena.
Traditional dense neural operators attempt to handle various physical simulations, such as fluid dynamics and porous media flows, within a single network structure. However, these different physical regimes impose incompatible demands on the model's parameters and geometry. This incompatibility can result in a "dense compromise" where the model fails to perform optimally across all simulated physics. Shodh-MoE aims to circumvent this by introducing modularity directly into the model's forward pass, rather than relying on external mechanisms or soft penalties.
The Shodh-MoE architecture begins with a physics-informed autoencoder that compresses physical data into 16x16x16 physical latents. This encoder incorporates a Helmholtz-style velocity parameterization, which ensures that decoded states remain within divergence-free velocity manifolds. This design choice guarantees exact mass conservation, achieving a verifiable velocity divergence of approximately 2.8 x 10^-10 on 128x128x128 grids.
Central to Shodh-MoE's operation is a Top-1 soft-semantic router. This component analyzes latent patches and directs them to specific expert subnetworks. This allows for specialized parameter paths for distinct physical mechanisms, while shared experts handle universal symmetries common across different physics domains. During a 20,000-step distributed pretraining run involving mixed three-dimensional physical tensors, the routing telemetry demonstrated an autonomous bifurcation. Validation tokens from open-channel fluid dynamics exclusively routed to Expert 0, while porous media flow tokens routed exclusively to Expert 1.
This specialized routing enabled the model to converge simultaneously across both regimes. The researchers reported latent validation mean squared errors (MSEs) of 2.46 x 10^-5 for one regime and 9.76 x 10^-6 for the other. Decoded physical MSEs were also low, at 2.48 x 10^-6 and 1.76 x 10^-6, respectively. These results suggest that sparse expert routing is an effective architectural strategy for mitigating interference between different physics simulations in universal neural operators.
The development of Shodh-MoE represents a step toward building more capable and scalable universal foundation models for scientific simulation. By moving physical constraints into architectural design and using routing to isolate conflicting physical dynamics, the model avoids the compromises inherent in dense architectures. The researchers believe this approach can serve as a foundational element for future multi-physics neural operators and generative inverse design applications.