Jakob Galley: Conservation Laws from Data Symmetry in Neural Networks

Abstract.

In this talk, we will explore whether intrinsic symmetries in training data lead to conserved quantities during gradient-flow training of neural networks.   We will show that under the assumption that the loss function is analytic and non-polynomial, data symmetries generically do not induce any additional conservation laws. For the mean squared error loss, on the other hand, there are situations in which data augmentation yields new conservation laws. We will present a framework we developed, which utilizes tensorizable networks to describe this phenomenon. Tensorizable networks are a family of architectures whose dependency on parameters and inputs can be separated through an intermediate representation. They include linear and polynomial networks, as well as Lightning Attention networks.   This presentation is based on joint work with Vahid Shahverdi and Axel Flinth (arXiv:2606.10913).