Wei Xu (Georgia Tech) “GPT-3 vs Humans: Rethinking Evaluation of Natural Language Generation” @ Hackerman Hall B17
Feb 24 @ 12:00 pm – 1:15 pm


While GPT models have shown impressive performance on summarization and open-ended text generation, it’s important to assess their abilities on more constrained text generation tasks that require significant and diverse rewritings. In this talk, I will discuss the challenges of evaluating systems that are highly competitive and perform close to humans on two such tasks: (i) paraphrase generation and (ii) text simplification. To address these challenges, we introduce an interactive Rank-and-Rate evaluation framework. Our results show that GPT-3.5 has made a major step up from fine-tuned T5 in paraphrase generation, but still lacks the diversity and creativity of humans who spontaneously produce large quantities of paraphrases.

Additionally, we demonstrate that GPT-3.5 performs similarly to a single human in text simplification, which makes it difficult for existing automatic evaluation metrics to distinguish between the two. To overcome this shortcoming, we propose LENS, a learnable evaluation metric that outperforms SARI, BERTScore, and other existing methods in both automatic evaluation and minimal risk decoding for text generation.


Wei Xu is an assistant professor in the School of Interactive Computing at the Georgia Institute of Technology, where she is also affiliated with the new NSF AI CARING Institute and Machine Learning Center. She received her Ph.D. in Computer Science from New York University and her B.S. and M.S. from Tsinghua University. Xu’s research interests are in natural language processing, machine learning, and social media, with a focus on text generation, stylistics, robustness and controllability of machine learning models, and reading and writing assistive technology. She is a recipient of the NSF CAREER Award, CrowdFlower AI for Everyone Award, Criteo Faculty Research Award, and Best Paper Award at COLING’18. She has also received funds from DARPA and IARPA. She is an elected member of the NAACL executive board and regularly serves as a senior area chair for AI/NLP conferences.

Paco Guzman (Meta AI) “Building a Universal Translation System to Break Down Language Barriers” @ Hackerman Hall B17
Apr 17 @ 12:00 pm – 1:15 pm


Machine Translation has the ultimate goal of eliminating language barriers. However, the area has focused mainly on a few languages, leaving many low-resource languages without support. In this talk, I will discuss the challenges of bringing translation support for 200 written languages and beyond.
First, I talk about the No Language Left Behind Project, where we took on this challenge by first contextualizing the need for low-resource language translation support through exploratory interviews with native speakers. Then, we created datasets and models aimed at narrowing the performance gap between low and high-resource languages. We proposed multiple architectural and training improvements to counteract over-fitting while training on thousands of language-pairs/tasks. We evaluated the performance of over 40,000 different translation directions.
Afterwards, I’ll discuss the challenges of pushing translation performance beyond text for languages that don’t have written standards like Hokkien.

Our models achieve state-of-the-art performance and lay important groundwork towards realizing a universal translation system. At the same time, we keep making open-source contributions for everyone to keep advancing the research for the languages they care about.


Paco is Research Scientist Manager supporting translation teams in Meta AI (FAIR). He works in the field of machine translation with a focus on low-resource translation (e.g. NLLB, FLORES) and the aim to break language barriers. He joined Meta in 2016. His research has been published in top-tier NLP venues like ACL, EMNLP. He was the co-chair of the Research director at AMTA (2020-2022). He has ave organized several research competitions focused on low-resource translation and data filtering. Paco obtained his PhD from the ITESM in Mexico, was a visiting scholar at the LTI-CMU from 2008-2009, and participated in DARPA’s GALE evaluation program. Paco was a post-doc and scientist at Qatar Computing Research Institute in Qatar in 2012-2016

Center for Language and Speech Processing