Abstract

As LLMs become increasingly integrated into academic workflows, their influence is both promising and precarious. In this talk, we will explore three facets of this evolving intersection.

– The Good: LLMs executing aspects of peer review that are difficult for human reviewers.

– The Bad: Vulnerabilities in the review process to fraud such as identity theft and collusion rings.

– The Ugly: Methodological pitfalls of autonomous “AI scientists.”

Bio

Nihar B. Shah is an Associate Professor in the Machine Learning and Computer Science departments at Carnegie Mellon University (CMU). His research is on the evaluation of science and the science of evaluation. His group develops computational tools with strong theoretical guarantees, and also designs and conducts controlled experiments for evidence-based policy design. His work has been used in the review of well over hundred thousand papers and thousands of proposals. He is a recipient of the Young Alumnus Medal from the Indian Institute of Science, a JP Morgan faculty research award, Google Research Scholar Award, an NSF CAREER Award 2020-25, the 2017 David J. Sakrison memorial prize from EECS Berkeley for a “truly outstanding and innovative PhD thesis”, the Microsoft Research PhD Fellowship 2014-16, the Berkeley Fellowship 2011-13, and several Best Paper Awards.