Home
About
People
Publications
Seminars
- Seminar Calendar
- Seminar Videos
Workshops
HLT Masters
Resources

Automatic Information and Lanuage Processing: Rethinking Evaluation

Add to Calendar

Add to Timely Calendar
Add to Google
Add to Outlook
Add to Apple Calendar
Add to other calendar
Export to XML

When:

August 11, 1999 all-day

1999-08-11T00:00:00-04:00

1999-08-12T00:00:00-04:00

Seminars

View Seminar Video

This post was replicated from another site's calendar feed. View original

Seminar Videos

Chenhao Tan (UChicago):
https://chenhaot.com/
https://cs.uchicago.edu/people/chenhao-tan/

Abstract
Human preferences have been the main component in guiding pretrained AI. In this talk, I argue that human preferences are often insufficient for complementing human intelligence and demonstrate the key role of human goals with two examples. First, hypothesis generation is critical for scientific discoveries. Instead of removing hallucinations, I will leverage data and labels as a guide to lead hallucination towards effective hypotheses. Second, I will use human perception as a guide for developing case-based explanations to support AI-assisted decision making. In both cases, faithfulness is “compromised” for achieving human goals. Finally, I will briefly discuss the case of identifying bias mechanisms in AI, where faithfulness is crucial. I will conclude with future directions towards complementary AI.

Bio
Tan is an assistant professor of computer science and data science at the University of Chicago, and is also affiliated with the Harris School of Public Policy. He obtained his PhD degree in the Department of Computer Science at Cornell University and bachelor’s degrees in computer science and in economics from Tsinghua University. Prior to joining the University of Chicago, he was an assistant professor at the University of Colorado Boulder and a postdoc at the University of Washington. His research interests include human-centered AI, natural language processing, and computational social science. His work has been covered by many news media outlets, such as the New York Times and the Washington Post. He also won a Sloan research fellowship, an NSF CAREER award, an NSF CRII award, a Google research scholar award, research awards from Amazon, IBM, JP Morgan, and Salesforce, a Facebook fellowship, and a Yahoo! Key Scientific Challenges award.

Beyond Human Preferences: Use Human Goals to Guide AI towards Complementary AI -- Chenhao Tan

Speaker: Jieyu Zhao (USC)
https://jyzhao.net/

Abstract: The rapid advancement of large language models (LLMs) has unlocked a myriad of possibilities for positive societal impact, ranging from enhancing accessibility and communication to supporting disaster response and public health initiatives. However, the deployment of these technologies also raises critical concerns regarding accountability, fairness, transparency, and ethical use. In this talk, I will discuss our efforts for auditing models, detecting and mitigating biases, and understanding how LLMs make decisions. We hope to open the conversation to foster a community-wide effort towards more accountable and inclusive LLM practices.

Trustworthy LLMs — our efforts on mitigating issues regarding social bias, safety and reliability

Abstract
Recent years saw the rise of multilingual language models that achieve high levels of performance for a large number of tasks, with some of them handling hundreds of languages at once. Premodern languages are usually underrepresented in such models, leading to poor performance in downstream applications. In my talk, I will introduce the Dharmamitra project, which aims to develop a diverse set of language models to address these shortcomings for the classical Asian low-resource languages Sanskrit, Tibetan, Classical Chinese, and Pali. These models are providing solutions for low-level NLP tasks such as word segmentation, morpho-syntactic tagging etc., as well as high-level tasks such as semantic search, machine translation, and general chatbot interaction. I will talk about the individual challenges and unique characteristics of the data involved, and what strategies we deploy to address these. I will also demonstrate how these different tools can be combined in an application that goes beyond simple sentence-to-sentence machine translation, but instead provides detailed grammatical explanations and corpus-wide search to provide the users with as much relevant information as possible. This application is helpful for early-stage languages learners on the one hand, as well as experienced researchers with high level of language knowledge and very specific demands on the other hand.

Biography
Sebastian Nehrdich is a research assistant at the University of California, Berkeley, where he is leading the MITRA project together with Prof. Kurt Keutzer. He is also currently completing a PhD degree in computational linguistics at the University of Duesseldorf, Germany. His main research interest is the application of NLP methods to premodern Asian languages with focus on Sanskrit, Pali, Tibetan and Classical Chinese. He holds a master’s degree in Buddhist Studies and worked extensively with literature in premodern languages while also publishing articles on NLP problems for these languages in major conferences. Since 2018, he is the chief developer of the BuddhaNexus platform, a website that provides an interactive and user-friendly interface for the exploration of textual reuse instances in large text collections of the Buddhist tradition with over a hundred million of intertextual links. Since 2023 The MITRA project provides grammatical analyzers, machine translation, semantic search capabilities for classical Asian languages and currently serves between 500-1000 users with primarily academic background on a daily basis.

MITRA: Beyond Just Machine Translation for Premodern Asian Low Resource Languages-Sebastian Nehrdich

Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirements (Jack Zhang)

Jingyu Jack Zhang: https://jackz.io/

Paper: https://arxiv.org/abs/2410.08968

Abstract: The current paradigm for safety alignment of large language models (LLMs) follows a one-size-fits-all approach: the model refuses to interact with any content deemed unsafe by the model provider. This approach lacks flexibility in the face of varying social norms across cultures and regions. In addition, users may have diverse safety needs, making a model with static safety standards too restrictive to be useful, as well as too costly to be re-aligned. We propose Controllable Safety Alignment (CoSA), a framework designed to adapt models to diverse safety requirements without re-training. Instead of aligning a fixed model, we align models to follow safety configs -- free-form natural language descriptions of the desired safety behaviors -- that are provided as part of the system prompt. To adjust model safety behavior, authorized users only need to modify such safety configs at inference time. To enable that, we propose CoSAlign, a data-centric method for aligning LLMs to easily adapt to diverse safety configs. Furthermore, we devise a novel controllability evaluation protocol that considers both helpfulness and configured safety, summarizing them into CoSA-Score, and construct CoSApien, a human-authored benchmark that consists of real-world LLM use cases with diverse safety requirements and corresponding evaluation prompts. We show that CoSAlign leads to substantial gains of controllability over strong baselines including in-context alignment. Our framework encourages better representation and adaptation to pluralistic human values in LLMs, and thereby increasing their practicality.

Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirements (Jack Zhang)

Abstract
Language model pretraining has been a remarkably strong recipe for cross-task and cross-domain generalization in NLP. However, these gains have come at the expense of control: we rarely control the training data for language models, and gaps between pretraining and our target evaluation lead to distribution shifts. We present two complementary approaches to control this gap — algorithmically filtering data to focus training on the most benchmark-relevant parts of the distribution, as well as adapting to new domains by synthesizing domain-specific pretraining data at scale.

Biography
Tatsunori Hashimoto is an Assistant Professor in the Computer Science Department at Stanford University. He is a member of the statistical machine learning and natural language processing groups at Stanford and studies statistical approaches to improving and understanding language models. Work from his group spans many areas, including instruction-following and controllable language models, differentially private fine-tuning, and benchmarks for LM safety and capabilities. He received his Ph.D. at MIT under the supervision of Tommi Jaakkola and David Gifford, and is the recipient of a Kavli fellowship, the NSF CAREER, faculty research awards from Google, Sony and Amazon, and best paper awards at ICML, ICLR, and CHI.

Improving LLM generalization by selecting and synthesizing data -- Tatsu Hashimoto (Stanford)

Load More... Subscribe

Johns Hopkins University, Whiting School of Engineering

Center for Language and Speech Processing
Hackerman 226
3400 North Charles Street, Baltimore, MD 21218-2680

(410) 516-4237
[email protected]

Whiting School of Engineering
Johns Hopkins University

Privacy Statement
Legal

© 2025 Johns Hopkins University

Center for Language and Speech Processing