ASR Machines That Know When They Do Not Know

Goal: Development of ASR systems that can successfully deal with new, unexpected data (“systems that know when they do not know” or “getting rid of unknown unknowns”)

To constrain the problem and provide resources for team members with different ideas, the core problem is stated as: Given a classifier that yields a frame-based vector of posterior probabilities for speech sounds of interest, predict the accuracy of these estimates without knowing the correct probabilities on test data but knowing performance of the classifier on the training data.

The main attack on this problem will be through multi-stream processing, where many parallel and partially redundant processing streams are derived from information providing data. This approach should be effective in many practical situations where the unexpected signal distortions negatively affect only some of the processing streams while the remaining streams can still be used for the extraction of the targeted information. The technique needs to be unsupervised, since the ground truth on the unknown data is not known, and fast, since new unexpected data need to be dealt with.

To date, research at JHU has resulted in formation of band-limited artificial neural net based processing streams for recognition of noisy speech, and in a couple of techniques for estimating the classifier performance based on temporal dynamics of classifier outputs. JHU will provide its multistream experimental system with 31 processing streams based on independent artificial neural net classifiers. Initial results on recognition of noise-corrupted TIMIT have been already obtained and will serve as a baseline. We will also provide the true accuracies for all processing streams, which would serve as the ideal targets of our efforts.

Team Members
Team Leader
Hynek Hermansky Johns Hopkins University
Senior Members
Lukas Burget Brno University of Technology
Jordan Cohen Spelamode Consulting
Naomi Feldman University of Maryland
Tetsuji Ogawa Waseda University
Richard Rose McGill University
Richard Stern Carnegie Mellon University
Graduate Students
Matthew Maciejewski Carnegie Mellon University
Harish Mallidi Johns Hopkins University
Anjali Menon Carnegie Mellon University
Vijayaditya Peddinti Johns Hopkins University
Matthew Wiesner McGill University
Affiliate Members
Eleanor Chodroff Johns Hopkins University
Emmanuel Dupoux Laboratoire de Science Cognitive et Psycholinguistique
John Godfrey Johns Hopkins University
Sanjeev Khudanpur Johns Hopkins University

Center for Language and Speech Processing