Home
About
People
Publications
Seminars
- Seminar Calendar
- Seminar Videos
Workshops
HLT Masters
Resources

WS’13 Summer Workshop

June 22, 2013 – July 24, 2013

Research Groups

Speaker and Language Recognition (June 22 – July 24)

Subscribe to the CLSP Seminars Mailing List

Upcoming Seminars

There are no upcoming events.

Add

Add to Timely Calendar
Add to Google
Add to Outlook
Add to Apple Calendar
Add to other calendar
Export to XML

Seminar Videos

CLSP Seminar given by Ting-Hao “Kenneth” Huang

Abstract:

Large language models (LLMs) have rapidly and impressively taken over many tasks once handled by human annotators in constructing text datasets. But does this mean we no longer need humans in the annotation loop? What roles should humans play in future data annotation pipelines? In this talk, I will present two recent studies that explore the evolving role of humans in the landscape of text data annotation. First, we ask whether a well-designed and carefully executed traditional crowdsourcing pipeline can still outperform LLMs in labeling quality. Our study offers an in-depth and holistic comparison between human and LLM annotation performance. Second, we turn to a future where LLMs increasingly replace manual annotation labor. In this scenario, the human role shifts toward instructing the models--often through prompting. But how effective are humans at prompting LLMs for annotation tasks, especially when working without access to gold-standard labels? We investigate this growing practice, which we call "prompting in the dark," and assess its implications for the quality and reliability of LLM-generated annotations.

Bio:

Dr. Ting-Hao 'Kenneth' Huang is an Associate Professor at the Pennsylvania State University's College of Information Sciences and Technology. With expertise in natural language processing (NLP) and human-computer interaction (HCI), he built interactive and intelligent systems that support people in achieving their social and creative goals in day-to-day activities. His systems generated captions for scientific figures (Best Long Paper of INLG 2023), enabled collective action that empowers community advocates to fight air pollution (Outstanding Paper Award of IUI 2019), conversed with hundreds of users about any topics well before ChatGPT (Best Paper Honorable Mention Award of CHI 2018), and aided creative writers in developing intricate story arcs. His research has been published in top-tier HCI, NLP, and AI conferences, including CHI, IUI, ACL, NAACL, EMNLP, HCOMP, and AAAI, and has been supported by Adobe, Meta, NSF, NIH, and Sloan Foundation. He co-founded and co-organizes the Workshop on Intelligent and Interactive Writing Assistants (In2Writing, https://in2writing.glitch.me/). Dr. Huang earned his Ph.D. in Computer Science from Carnegie Mellon University in 2018. More about his work can be found at http://kennethhuang.cc/

What Roles Will Humans Play in the Future of Data Annotation?

CLSP Seminar Talk by Niyati Bafna

Abstract:

While state-of-the-art models for machine translation and natural language understanding perform well on high-resource members of (some of) the world’s language families, they degrade on closely-related languages, dialects, and variants of these languages, which are typically much lower-resourced. However, a considerable part of the divergence between related languages is largely systematic and well-understood. In this talk, we first set up a framework within which we can understand performance degradation on a low-resource language as a function of linguistic distance from a high-resource neighbour, using artificial dialects generated at controlled distances from a given language.

Next, we introduce DialUp, a method to induce robustness to the dialect continua of training languages in pretrained machine translation models, including for unseen dialects. We discuss the success of our approach as depending on various factors, and briefly talk about next directions in this space.

Evaluating and Inducing Dialectal Robustness in Large Language Models

CLSP Seminar talk by Igor Molybog, PhD, Assistant Professor at the University of Hawaii at Manoa.

Talk Abstract:

In this talk, Igor will share the latest findings from his research group on the development, evaluation, and deployment of large language models (LLMs) and their multi-modal extensions. We will outline the end-to-end pipeline, with a particular focus on the challenges and costs associated with data generation in the later stages of training. Additionally, we will explore strategies for integrating multiple models into inference systems to enhance efficiency. Finally, we will highlight key tasks where even the most advanced multi-modal models face difficulties and discuss approaches that yield significantly better performance.

Speaker Bio:

Igor is an Assistant Professor at the University of Hawai'i at Manoa. In 2022, I graduated with a Ph.D. in Engineering from UC Berkeley and worked as a research engineer at Google DeepMind and Meta AI. He specializes in large language models (LLM), focusing on the efficient development and robust evaluation of computer systems that automate economically viable yet tedious tasks typically requiring human intervention. His immediate areas of interest include:

Expanding AI's Impact: Identifying novel use cases for automation with LLM and novel tasks for increasing professionals’ productivity. Developing frameworks for robust evaluation of AI agents. Sourcing diverse data to address unique challenges in emerging applications.
Multimodal Modeling: Integrating video and other sensory modalities into AI reasoning to advance robotics and other cyber-physical systems applications. Enhancing large language models’ capabilities in computer programming and mathematical problem-solving.
Core Machine Learning and Scaling: Addressing efficiency challenges and fundamental obstacles to expand the computational resources available to AI systems. Optimizing the design of modeling experiments and developing predictable training processes.

Development and deployment of large-scale AI systems and tasks they still struggle with

Tianjian Li: https://tianjianl.github.io/

Related paper: https://arxiv.org/abs/2410.04579

Abstract: Data abundance across different domains exhibits a long-tailed distribution: few domains have abundant data, while most face data scarcity. Our work focuses on a multilingual setting, where available data is heavily skewed towards high-resource languages. Two common strategies to address this disparity are upsampling low-resource data (Temperature Sampling) and upweighting low-resource loss (Scalarization). These methods are often assumed to be equivalent, but this equivalence has not been rigorously established, prompting our investigation.
Through theoretical and empirical analysis, we identify when these two methods are equivalent and when they diverge. We prove that they are equivalent under full gradient descent but differ under stochastic gradient descent due to differences in gradient variance. Specifically, Temperature Sampling exhibits lower variance in gradient estimation compared to Scalarization, leading to faster convergence but a higher risk of overfitting. Based on these insights, we propose Cooldown, a strategy that starts by heavily upsampling low-resource languages to accelerate convergence and gradually reduces the upsampling to prevent overfitting -- achieving the best of both worlds. Our method competes effectively with existing data re-weighting techniques while offering computational efficiency.

Upsample or Upweight? Balanced Training on Heavily Imbalanced Datasets -- NAACL 2025

Based on the paper: https://arxiv.org/abs/2404.03862
Speaker: Jingyu (Jack) Zhang https://jackz.io/
Abstract:
To trust the fluent generations of large language models (LLMs), humans must be able to verify their correctness against trusted, external sources. Recent efforts, such as providing citations via retrieved documents or post-hoc provenance, enhance verifiability but provide no guarantees on their correctness. To address these limitations, we tackle the verifiability goal with a different philosophy: trivializing the verification process by developing models that quote verbatim statements from trusted sources in their pre-training data. We propose Quote-Tuning, which demonstrates the feasibility of aligning models to quote. The core of Quote-Tuning is a fast membership inference function that efficiently verifies text against trusted corpora. We leverage this tool to design a reward function to quantify quotes in model responses, and curate datasets for preference learning. Experiments show that Quote-Tuning significantly increases verbatim quotes from high-quality documents by up to 130% relative to base models while maintaining response quality. Quote-Tuning is applicable in different tasks, generalizes to out-of-domain data and diverse model families, and provides additional benefits to truthfulness. Our method not only serves as a hassle-free method to increase quoting but also opens up avenues for improving LLM trustworthiness through better verifiability.

Verifiable by Design: Aligning Language Models to Quote from Pre-Training Data (NAACL 2025)

Load More... Subscribe

Johns Hopkins University, Whiting School of Engineering

Center for Language and Speech Processing
Hackerman 226
3400 North Charles Street, Baltimore, MD 21218-2680

(410) 516-4237
[email protected]

Whiting School of Engineering
Johns Hopkins University

Privacy Statement
Legal

© 2025 Johns Hopkins University

Center for Language and Speech Processing