Abstract

Recently, we have seen tremendous progress in the field of language models (LMs), with the release of numerous open models and closed API systems. However, fewer and fewer disclose how they are created: Which corpora do they use? How are they trained? How much energy do they consume? In this talk, I will provide an overview of OLMo (https://allenai.org/olmo), an initiative at AI2 aimed at creating transparent artifacts and tools that advance the science of LMs. I will discuss current releases, such as Tulu, Dolma, and OLMo-7b, as well as the goals, ethical, and legal considerations of this initiative, and what’s coming next.

Biography

Luca Soldaini (https://soldaini.net) is a Senior Applied Research Scientist at the Allen Institute for AI, where they co-lead the pre-training data team for OLMo. Their current research focuses on data-centric NLP, information retrieval, and the use of LMs for scientific applications. Prior to joining Ai2 in 2022, Luca was a Senior Applied Scientist at Amazon Alexa, where they worked on Open Domain Question Answering. Luca obtained their PhD from Georgetown University in 2018.