Adam Byerly (JHU) “How Effective Is Self-Consistency for Long-Context Problems?”
3400 N CHARLES ST
Baltimore
MD 21218
Abstract
Self-consistency (SC) has been demonstrated to enhance the performance of large language models (LLMs) across various tasks and domains involving short content. However, does this evidence support its effectiveness for long-context problems? In this talk, we examine the role of SC in long-context scenarios, where LLMs often struggle with position bias, hindering their ability to utilize information effectively from all parts of their long input context. We examine a range of design parameters, including different models, context lengths, prompt formats, and types of datasets and tasks. Our findings reveal that, although SC is effective for short-context problems, it does not mitigate position bias and can sometimes hurt overall performance. We observe that the effectiveness of SC varies with context length and model size but remains mainly unaffected by prompt format or task type. These results provide valuable insight into the limitations of current LLMs in long-context understanding and highlight the need for more sophisticated approaches to address position bias in these models.