Johanna Marie Gegenfurtner: The Symmetries of Three-Layer ReLU Networks

Abstract.

We develop a framework for analyzing parameter symmetries in deep ReLU networks and obtain a complete characterization of the generic parameter fibers for
three-layer bottleneck architectures. Our approach provides explicit semi-algebraic
descriptions of these fibers and yields a polynomial time algorithm for deciding
functional equivalence of two parameters. The symmetries include discrete and
continuous transformations arising from layer composition, and depend on whether
deeper layers hide or preserve geometric structure from preceding layers. Finally,
we show that some of these symmetries induce local conservation laws along
gradient flow, while others do not.