Abstract

Any valuable NLP dataset has traditionally been shipped with crowdsourced categorical labels. Instructions for collecting these labels are easy to communicate and the labels themselves are easy to annotate. However, as self-supervision based methods are getting better at basically everything, human annotations may need to provide more nuanced supervision or enable more detailed evaluation in order to be worth further collecting. One natural extension to existing categorical annotation schemes is to obtain uncertainty information beyond a single hard label. In this talk, I will discuss my recent efforts on introducing scalar labels in place of categorical labels as a form of uncertainty annotation. We demonstrate that, compared to other more obvious annotation schemes for eliciting uncertainty information, scalar labels are significantly more cost-effective to annotate, provide reliable evaluation, and have a theoretical connection to existing predictive uncertainty metrics. In particular, they motivate using other losses as surrogates for calibration evaluation.