In the last decade, deep learning models have become an integral part of our lives, from image recognition software to large language models such as ChatGPT. These models are often overparameterized, with many (many!) more parameters than training examples. While this naively implies that these models should just memorize their training data, instead, we find that they generalize extremely well, finding solutions that are often even better than more traditional underparameterized models. This almost-magical ability to generalize rather than memorize turns out to be (in part) the result of how we optimize and sample from these overwhelmingly large models’ parameters. Motivated by this behaviour, this project will investigate new variants of optimization and/or sampling methods based on ideas from Hamiltonian optimization, dynamics, and optics, to see how well they perform across a wide class of problems (potentially including large language models). This will involve a combination of theoretical work as well as empirical studies of how these methods perform on various methods on benchmarks, their stability and dynamics under various conditions, and the implicit and/or explicit regularization that they provide.This project will be co-supervised with Prof. Ricardo Baptista, Department of Statistical Sciences, Faculty of Arts & Science, University of Toronto. 

The SUDS Scholar will be actively involved in pursuing a combination of both (1) theoretical work and literature review, as well as (2) conducting empirical studies of how these methods perform (including their stability, dynamics, and regularization properties) across various benchmarks.