A new approach to estimating Gaussian Mixture Models (GMMs), which frequently model complex data distributions in machine learning and statistics, has been developed by Romane Giard, Yohann De Castro, Roland Denis, and Clément Marteau. Their method, detailed in a paper on arXiv, addresses the challenges of GMM estimation when the number of components and their covariance matrices are unknown. The researchers combined the Conic Particle Gradient Descent (CPGD) principle with Riemannian gradient descent, which accounts for the Fisher-Rao geometry inherent to Gaussian distributions.
The research focuses on the numerical resolution of the Beurling-LASSO (BLASSO), a convex optimization framework designed to promote sparsity in the space of measures. The application of BLASSO to GMMs allows for simultaneous estimation of the component count and their parameters. The authors highlight two primary contributions: providing theoretical guarantees for the algorithm's convergence and presenting numerical experiments that illustrate its performance.
The theoretical guarantees include establishing exponential local convergence under a non-degeneracy condition on the solution. This condition is linked to a separation condition on the underlying statistical target, meaning the algorithm's rapid convergence is dependent on the distinctness of the Gaussian components.
For practical implementation, the algorithm uses a particle-based discretization. This involves iteratively updating both the weights and positions of these particles. The weight updates employ a conic retraction step to ensure non-negativity, while position updates leverage the Fisher-Rao geometry through a Riemannian Gradient Descent scheme. This scheme is based on the Fisher-Rao metric tensor and updates follow a geodesic path, implemented via Natural Gradient Descent with AdaGrad adaptive step sizes.
Numerical experiments conducted by the researchers suggest that their CPGD-based approach is more robust to overspecification of the number of components compared to the widely used Expectation-Maximization (EM) algorithm. The EM algorithm is known to be sensitive to the initial number of components specified, often converging to local maxima due to the non-convex nature of the log-likelihood function.
The study also investigated the impact of component separation on recovery accuracy. The experiments revealed a "phase transition" in accuracy. When the separation between components was insufficient relative to their variance, the performance degraded sharply. In such high-overlap scenarios, increasing the sample size provided negligible gains, indicating a fundamental statistical limit.
This work extends the BLASSO methodology to multivariate GMMs with component-specific unknown diagonal covariance matrices, offering greater flexibility than prior approaches that required known and identical covariances. The ability to estimate GMMs without prior knowledge of the number of components or their covariance structures has implications for various applications, including clustering, density estimation, image processing, bioinformatics, and economics.
The research was conducted by Romane Giard from Centrale Lyon and Université Lyon 1, Yohann De Castro from INSA Lyon and Université Lyon 1, Roland Denis from Université Jean Monnet and CNRS, and Clément Marteau from Université Lyon 1 and Institut Universitaire de France.