Abstract

Humans have an impressive ability to understand long and complex narratives, and to utilize common sense knowledge to quickly comprehend novel situations. NLP systems tend to scale poorly to long texts, and to rely on extended batch training before being able to make inferences. In this talk, I will present two projects aimed towards improving automatic understanding and modeling of long and complex narratives.

The first part presents work on leveraging topical document structure for improved and more flexible summarization. Given a topic and a news article, our topic-aware summarization models, summarize the article with respect to the topic. I will present a scalable synthetic training setup, and show that modeling document structure is particularly useful for long documents.

The second part of the talk focuses on incremental inference in a complex, multi-modal, and evolving world, considering the task of incremental identification of the perpetrator in episodes of a TV crime series (CSI). I will present a model, data set, task formulation and extensive analysis of the quality of model predictions compared to human predictions for the same task.

Biography

Lea is a postdoc at Amazon Core AI (Berlin), currently spending a 5 week visit at Columbia University, New York. In July 2019 she will take up a lecturer position at Melbourne University. Previously, she was a research associate at the University of Edinburgh, and a visiting scholar at Stanford. She obtained a PhD from the University of Edinburgh in 2017 (supervised by Mirella Lapata). Her research investigates the efficiency and robustness of human learning and inference in the face of the complexity of the world as approximated, for example, through large corpora of child-directed speech, or plots of books and films.

Abstract

Traditional semantic representation frameworks generally define complex, often exclusive category systems that require highly trained annotators to build. And in spite of their high quality for the cases they are designed to handle, these frameworks can be brittle to cases that (i) deviate from prototypical instances of a category; (ii) are equally good instances of multiple categories; or (iii) fall under a category that was erroneously excluded from the framework’s ontology.

I present an alternative approach to semantic representation that addresses these issues, under the auspices of the Decompositional Semantics Initiative. In this approach, which is rooted in a long tradition of theoretical approaches to lexical semantics, semantic representations are decomposed into a straightforwardly extensible set of core inferences about entities and events that any native speaker can recognize—e.g. inferences about the animacy of an entity referred to by a noun phrase or the the factuality and temporal duration of an events referred to by a predicate. A consequence of this approach is that semantic annotation can take the form of many simple questions about words or phrases (in context) that are easy for naive native speakers to answer, thus allowing annotations to be crowd-sourced while retaining high interannotator agreement.

I discuss five datasets that apply this decompositional approach to five domains—semantic roles, event factuality, genericity, temporal relations, and entity typing—along with a range of models aimed at predicting decomposed representations of these phenomena. I then present a recently released graph bank—Universal Decompositional Semantics v1.0 (UDS1.0)—that unifies these datasets into a single semantic graph representation, annotated with real-valued node and edge attributes, as well as a parser that can predict these graphs and their annotations from raw text.

Biography

Aaron is an Assistant Professor in the Department of Linguistics at the University of Rochester, with secondary appointments in the Department of Computer Science and the Department of Brain and Cognitive Sciences and an affiliation with the Goergen Institute for Data Science. He is also the Director of the FACTS.lab at UR, and co-leads two multi-institution projects: the MegaAttitude Project and the Decompositional Semantic Initiative. Before joining the University of Rochester, he received his PhD in Linguistics from the University of Maryland in 2015 and was a postdoctoral fellow at Johns Hopkins University’s Science of Learning Institute with affiliations in the Department of Cognitive Science and the Center for Language and Speech Processing from 2015 to 2017. Aaron’s research focuses on issues of semantic representation and natural language ontology, both in humans and machines. It aims to understand how humans represent the meanings of words, how those meanings relate to the meanings of the syntactic structures those words occur in, and how the nature of these representations can inform the way natural language understanding systems are built.

Abstract

With all of the modeling advancements in recent years, NLP benchmarks have been falling over left and right: “human performance” has been reached on SQuAD 1 and 2, GLUE and SuperGLUE, and many commonsense datasets.  Yet no serious researcher actually believes that these systems understand language, or even really solve the underlying tasks behind these datasets.  To get benchmarks that we actually believe in, we need to both think more deeply about the language phenomena that our benchmarks are targeting, and make our evaluation sets more rigorous.  I will first present ORB, an Open Reading Benchmark that collects many reading comprehension datasets that we (and others) have recently built, targeting various aspects of what it means to read.  I will then present contrast sets, a way of creating non-iid test sets that more thoroughly evaluate a model’s abilities on some task, decoupling training data artifacts from test labels.

Biography

Matt is a senior research scientist at the Allen Institute for AI on the AllenNLP team. His research focuses primarily on getting computers to read and answer questions, dealing both with open domain reading comprehension and with understanding question semantics in terms of some formal grounding (semantic parsing). He is particularly interested in cases where these two problems intersect, doing some kind of reasoning over open domain text. He is the original architect of the AllenNLP toolkit, and he co-hosts the NLP Highlights podcast with Waleed Ammar and Pradeep Dasigi.