Rebecca Steorts (Duke University) “Flexible Models for Microclustering with Application to Entity Resolution”

November 18, 2016 @ 12:00 pm – 1:15 pm
Hackerman Hall B17
3400 N Charles St
Baltimore, MD 21218
Center for Language and Speech Processing


Record linkage merges together large, potentially noisy databases to remove duplicate entities. Community detection is the process of placing entities into similar partitions or “communities.” Both applications are important to applications in author disambiguation, genetics, official statistics, human rights conflict, and others. It is common to treat record linkage and community detection as clustering tasks. In fact, generative models for clustering implicitly assume that the number of data points in each cluster grows linearly with the total number of data points. Finite mixture models, Dirichlet process mixture models, and Pitman–Yor process mixture models make this assumption. For example, when performing record linkage, the size of each cluster is often unrelated to the size of the data set. Consequently, each cluster contains a negligible fraction of the total number of data points. Such tasks require models that yield clusters whose sizes grow sublinearly with the size of the data set. We address this requirement by defining the “microclustering property” and discussing a new model that exhibits this property. We illustrate this on real and simulated data.

This is joint work with Giacomo Zanella Brenda Betancourt, Jeff Miller, Hanna Wallach, and Abbas Zaidi

Rebecca C. Steorts, Duke University,  Assistant Professor, Departments of Statistical Science and Computer Science; affiliated faculty in Computer Science, Biostatistics, the Social Science Research Institute, and the information initiative at Duke


Center for Language and Speech Processing