Abstract

Social media allows researchers to track societal and cultural changes over time based on language analysis tools. Many of these tools rely on statistical algorithms which need to be tuned to specific types of language. Recent studies have questioned the robustness of longitudinal analyses based on statistical methods due to issues of temporal bias and semantic shift. To what extent are changes in semantics over time affecting the reliability of longitudinal analyses? We examine this question through a case study: understanding shifts in mental health during the course of the COVID-19 pandemic. We demonstrate that a recently-introduced method for measuring semantic shift may be used to proactively identify failure points of language-based models and improve predictive generalization over time. Ultimately, we find that these analyses are critical to producing accurate longitudinal studies of social media.