Abstract

In defining “language understanding” for the purposes of natural language processing (NLP), we must inevitably be informed by human cognition: the only existing system that has achieved language understanding. Use of human cognition to evaluate NLP systems is nothing new – nearly any NLP benchmark relies on some form of comparison to human judgments or productions. In this talk I will discuss a series of projects taking this rationale a step further, examining NLP systems’ capturing of information by drawing on our knowledge of information sensitivity at a number of different levels of human cognition. Ideally we want our systems to extract and represent the same information that humans do at the endpoint of language comprehension – and because we have an idea of what that information is, we can test for it accordingly. However, we find at times that the representational patterns observed in our NLP systems show parallels instead with earlier stages of human language processing that reflect coarser information sensitivity. I discuss experiments examining both of these types of parallels: tests probing the extent to which NLP representations reflect the compositional meaning information to be expected in the final representation of a sentence, as well as tests examining the correspondence of NLP representations with earlier stages of human comprehension reflected in human predictive brain responses. I discuss the implications of these results for our assessment of these models, and for our targeting of ideal representational capacities as we improve our models moving forward.

Bio

Allyson Ettinger is an Assistant Professor in the Department of Linguistics at the University of Chicago. Her interdisciplinary work draws on methods and insights from cognitive science, linguistics, and computer science to examine the extraction, representation, and deployment of meaning information during language processing in humans and NLP systems. She received her PhD in Linguistics from the University of Maryland, and spent one year as research faculty at the Toyota Technological Institute at Chicago (TTIC) before beginning her appointment at the University of Chicago. She holds an additional courtesy appointment at TTIC.