Talk at MORTech 2025

Natural gradient descent (NGD) recently received a lot of attention for approximating functions in nonlinear manifolds [1]. NGD can be seen as a preconditioned update where parameter changes are driven by a functional perspective [2]. In a spirit similar to a second order (or Newton’s) method, the NGD update uses, instead of the Hessian, the Gram matrix of the generating system of the tangent space to the approximation manifold at the current iterate, with respect to a suitable metric [3]. Although the assemblage and inversion of the Gram matrix is prohibitively expensive in the context of big machine learning models, it becomes not only feasible but necessary when we look at scientific machine learning problems [4] not requiring as many parameters. For example when searching for the solution of a parametric partial differential equation (pPDE) by solving the forward (given parameters) or inverse (given measurements) problems using Physics Informed Neural Networks (PINNs) traditional ubiquitous solvers like Adam or L-BFGS do not yield reliable solutions or they take too long to converge [5]. However, taking the natural gradient perspective allows us to reinterpret gradient descent as a projection of the functional gradient into the tangent space of the approximation manifold obtaining great improvements. That being said, both gradient and natural gradient descent will still get stuck at any local minima. Furthermore, when the loss function is other than the L2 distance (for example when minimising the residual of a pPDE as in PINNs) even the natural gradient might yield non-optimal directions at each step. We will first focus on how we can tackle these situations by introducing a Natural version of classical inertial dynamic methods like Nestorov [6] or heavy-ball [7] and second we will show how this strategy can be used to improve the optimization of PINNs.

Nov 27, 2025 12:00 PM