Abstract

Legal professionals need to write analyses that rely on citations to relevant precedents, i.e., previous case decisions. Intelligent systems assisting legal professionals in writing such documents provide great benefits but are challenging to design. Such systems need to help locate, summarize, and reason over salient precedents in order to be useful. To enable systems for such tasks, we work with legal professionals to transform a large open-source legal corpus into a dataset supporting two important backbone tasks: information retrieval (IR) and retrieval-augmented generation (RAG).  We benchmark state-of-the-art models, showing that current approaches still struggle: GPT-4o generates analyses with the highest ROUGE F-scores but hallucinates the most, while zero-shot IR models only achieve 48.3% recall@1000. In a subsequent project, we also look at involved ways of evaluating legal hallucination with an expert-guided taxonomy and detector.

Bio

Abe Hou is a senior undergraduate student at JHU, working with Benjamin Van Durme and Daniel Khashabi. He is interested in Legal NLP, AI Policy/Governance, and applications of LLM agents in public domains.