Two trends have taken hold in machine learning and artificial intelligence: a move to massive, general-purpose, pre-trained models as well as a move to small, on-device models trained on distributed data. Both these disparate settings face some common challenges: a need for (a) robustness to deployment conditions that differ from training, (b) faster optimization, and (c) protection of data privacy.
As a result of the former trend, large language models have displayed emergent capabilities they have not been trained for. Recent models such as ChatGPT have attained the ability to generate remarkably human-like long-form text. I will describe Mauve, a measure to quantify this ability by measuring the gap between the distribution of generated text and that of human-written text. I will highlight its good empirical performance and present some statistical estimation results.
The move to massively distributed on-device federated learning of models opens up new challenges due to the natural diversity of the underlying user data and the need to protect its privacy. I will discuss how to reframe the learning problem to make the model robust to natural distribution shifts arising from deployment on diverse users who do not conform to the population trends in a manner that admits a distributed optimization algorithm with end-to-end differential privacy.
To conclude, I will discuss my ongoing efforts and future plans to work toward the next generation of ML/AI techniques by combining the best of both worlds. I will discuss applications ranging from differentially private language models and text generation to decentralized learning.
Bio: Krishna Pillutla is a visiting researcher (postdoc) at Google Research, USA in the Federated Learning team. He obtained his Ph.D. at the University of Washington where he was advised by Zaid Harchaoui and Sham Kakade. Before that, he received his M.S. from Carnegie Mellon University and B.Tech. from IIT Bombay where he was advised by Nina Balcan and J. Saketha Nath respectively. Krishna's research has been recognized by a NeurIPS outstanding paper award (2021) and a JP Morgan Ph.D. fellowship (2019-20).
