Jan Gerken: Emergent Equivariance in Deep Ensemble

Abstract.

In this talk, I will discuss recent results on the symmetry properties of deep ensembles trained with data augmentation. Specifically, we show that the ensemble is equivariant at any training step, provided that data augmentation is used. Crucially, this equivariance also holds off-manifold and therefore goes beyond the intuition that data augmentation leads to approximately equivariant predictions. Furthermore, equivariance is emergent in the sense that predictions of individual ensemble members are not equivariant but their collective prediction is. Therefore, the deep ensemble is indistinguishable from a manifestly equivariant predictor. In the infinite width limit, this predictor is in fact a group convolutional neural network. We prove this theoretically using neural tangent kernel theory and verify our theoretical insights using detailed numerical experiments. Based on joint work with Pan Kessel and Philipp Misof: arXiv:2403.03103 and arXiv:2406.06504.