Abstract

Open source generative models like OpenGPT2, BLOOM, and others have been pivotal in advancing AI technology. These models leverage extensive text data to achieve advanced linguistic capabilities. However, the trend towards proprietary tools and closed large language models is growing, posing unique challenges in open-source AI development. This discussion will explore the challenges and opportunities training these foundation models, the hurdles in dataset governance, and downstream AI for science applications. We will also explore algorithmic improvements in training these models such as discrete diffusion, learned adaptive noise schedules, and modern GAN baselines. We will cover the challenges of generative models in several different modalities: text, image, and biological sequence data.

Bio

Aaron Gokaslan is a 4th-year PhD Candidate at Cornell University, advised by Volodymyr Kuleshov. He collaborates closely with James Grimmelmann, Noah Snavely, and Sasha Rush. Prior to his PhD, Aaron worked at Facebook AI Research under the guidance of Dhruv Batra. He completed his undergraduate and master’s degrees at Brown University, where he worked with James Tompkin. Aaron’s research focuses on identifying, designing, and building efficient, scalable, sustainable, and affordable abstractions and infrastructure for generative modeling research. He also works at the intersection of data, law, and AI policy. His contributions have been recognized through orals and invited talks at top conferences such as NeurIPS, ECCV, ICML, and CVPR. A Mozilla RISE25 2024 honoree, Aaron has received awards for his open-source contributions from the Linux Foundation and Mozilla. He maintains popular open-source libraries like pybind11 and PyTorch. He was among the first to release a 1-billion-parameter+ auto-regressive large language model, OpenGPT2. Aaron has also contributed to the development of widely-used generative AI artifacts, including OpenWebText, OpenGPT2, CommonCanvas, CommonCatalog, CommonPile, Habitat-Matterport3D, DataComp-LM, and BLOOM, which have collectively been downloaded millions of times. He also serves on the advisory boards of Fidutam and Encode Justice.