Séminaire exceptionnel (Susan WEI, jeudi 13 juin)

Un séminaire exceptionnel aura lieu jeudi13 juin à 14h en salle A411. Susan WEI (University of Melbourne) nous parlera de "What's Degeneracy Got to Do with It? Understanding Deep Neural Networks through the Local Learning Coefficient".

Abstract
Deep neural networks (DNN) are singular statistical models that exhibit complex degeneracies. In this work, we introduce a quantity known as the Local Learning Coefficient (LLC) which precisely quantifies the degree of degeneracy in DNNs. Although the LLC is designed to address the limitations of traditional complexity measures, it coincides with familiar notions of complexity when the model is regular or minimally singular. We introduce a scalable estimator for the LLC and apply it across diverse DNN architectures including deep linear networks up to 100M parameters, ResNet image models, and transformer language models. Empirical evidence suggests that the LLC provides valuable insights into how common deep learning training heuristics might influence the effective complexity of DNNs. Ultimately, the LLC emerges as a valuable tool for reconciling the apparent contradiction between deep learning's complexity and the principle of parsimony.

This is joint work with my PhD students Edmund Lau and Zach Furman, and colleague Dr. Daniel Murfet.