Marco Nurisso: Conservation Laws, Connectivity, and Singularities in the Learning Space of ReLU Networks

Abstract.

Understanding the geometry and topology of the parameter space of feed-forward ReLU networks is critical for understanding training dynamics. A fundamental observation is that gradient flow training preserves a family of conservation laws arising from the homogeneous nature of the ReLU activation function, which decisively restricts the reachable parameter space to a lower-dimensional algebraic variety.

In this talk, I will present a study of two key structural features of this variety for networks built on general directed acyclic graph (DAG) architectures: the (dis)connectedness of the learning space and the existence and nature of its singularities. I will give a thorough characterization of connectedness, highlighting the roles of bottleneck neurons and balance conditions associated with specific subsets of the network. I will show how singularities are strongly tied to the topology of the underlying DAG and its induced sub-networks, discuss their reachability, and establish a principled connection with differentiable pruning.