Abstract

Current Natural Language Processing systems rely heavily on Large Language Models (LLMs) which are trained rather naively on large amounts of text using autoregressive next token prediction. Often, they are fine-tuned to mimic human preferences in order to accomplish more complex tasks. While these methods work surprisingly well, they have inherent biases stemming from design and engineering decisions arising from focusing on monolingual English. In this talk, I’ll discuss improved algorithms that we have developed to overcome deficiencies unearthed by working on multilingual and translation tasks. These improvements are general enough to be applied back to standard monolingual English training and fine-tuning. For instance, our CPO algorithm uses half the GPU memory of other popular Preference Optimization methods, or our Cooldown Temperature Sampling algorithm for Pre-Training converges faster and prevents overfitting. By focusing on languages beyond English, this talk will cover how we can gain insights into developing more efficient and robust general methods.

Bio

Kenton Murray is a Research Scientist at Johns Hopkins University where he directs the Maieutic Lab. His work focuses on Multilingual Natural Language Processing and Machine Translation with a particular emphasis on low-resource and morphologically rich languages across text, speech, and vision modalities. He received his PhD from the University of Notre Dame, a Master’s from Carnegie Mellon University, and a Bachelor’s from Princeton University.