Benjamin Ruppik: Measuring the Topology of Word Embeddings

Abstract.

Understanding the internal mechanisms of large language models (LLMs) remains a challenging and complex endeavor. Even fundamental questions, such as how fine-tuning affects model behavior, often require extensive empirical evaluation. In our latest work, we introduce a novel perspective based on the geometric properties of contextual latent embeddings to study the effects of training and fine-tuning. To that end, we measure the local intrinsic dimensions of an LLM's latent space and analyze their shifts, which provide insights into the model's training dynamics and generalization ability. Our experiments suggest a practical heuristic: reductions in the mean local dimension tend to accompany and predict subsequent performance gains.

The talk is primarily based on our paper "Less is More: Local Intrinsic Dimensions of Contextual Language Models" which will be presented at NeurIPS 2025 (https://arxiv.org/abs/2506.01034), but will also cover broader applications of Topological Data Analysis in machine learning.