Archived Seminars by Year

Abstract

A typical large vocabulary automatic speech recognition (ASR) system consists of three main components: 1) the feature extraction, 2) the pattern classification, and 3) the language modeling. Replacing hardwired prior knowledge in the pattern classification and language modeling modules by the knowledge derived from the data turned out to be one of most significant advances in ASR research in past two decades. However, the speech analysis module so far resisted the recent data-oriented revolution and is typically built on textbook knowledge of speech production and perception. Our current research aims at extending the data-driven notion to speech analysis. Since speech was optimized by millennia of human evolution to serve its purpose of communication through imperfect production-environment-perception channel, it carries imprints of the channel. It could, therefore, be gratifying but would come as no surprise if the data-driven analysis yields solutions consistent with properties of human speech production and perception. In the talk we first describe our efforts to use the concept of mutual information used on a relatively large phoneme hand-labeled database of fluent speech to estimate distribution of information which is most relevant for phoneme classification in the time-frequency plane. We demonstrate that this information is distributed over a significant time interval around the given phoneme. The Linear Discriminant Analysis is then used to derive optimized spectral basis functions and filters (replacing conventional cosines of the cepstral analysis and conventional RASTA and delta filters for deriving dynamic features) in processing the time-frequency plane of the speech signal. The last part of the talk describes our initial efforts for departing from the conventional assumption of importance of across-spectrum features (as is e.g. cepstrum) and towards frequency localized classifiers of relatively long (about 1-sec) temporal patterns of critical-band spectral energies. Work supported by the Department of Defense and by the National Science Foundation.  

Abstract

A new generation of systems is emerging in which the user is able to employ natural communication modalities, including speech and pen-based gesture, in addition to the usual graphical user interface technologies. Multimodal systems incorporating pen and voice communication are advantageous for both very small and very large devices, for spatially-oriented applications, and for contexts emphasizing user mobility. These advantages will be illustrated through QuickSet -- a handheld, collaborative, multimodal system that allows continuous speech and pen-based gesturing as input. QuickSet uses a distributed agent architecture, runs on personal computers, and is scalable from wearable to wall-sized systems. Among QuickSet's applications are initializing military simulations, control of virtual reality environments, logistics planning, and medical informatics. The core of QuickSet is a principled method for combining information derived from different modes. We discuss how a set of meaning fragments produced by recognizers for multiple modes can be unified to determine the best joint interpretation. This unification process will be shown to support multimodal discourse and mutual disambiguation of those meaning fragments. Finally, to assess the impact of multimodal interaction, a study will be described in which expert users completed map-based military tasks using both a graphical user interface and QuickSet. In brief, with the multimodal interface, users positioned entities on a map 3 - 8 times faster than with the graphical user interface. Multimodal interaction was preferred by all users, particularly for its efficiency and for its precision in drawing. To illustrate the QuickSet technology and its applications, a video and demonstration of the system will be given.

Abstract

We examine image understanding from the classical source-channel point of view of statistical communications. The space of images corresponding to the source is a Grenander deformable template, an orbit under the group action of diffeomorphisms of a prototype. The prior distribution on the source is induced through a distribution on the group. The channel corresponding to the remote sensor generates the observable images reflecting projection and noise which is statistically modeled via a conditional probability density, the likelihood function. Minimum-risk estimation, rate-distortion, and compression are examined by introducing a distance between images via a distance on the group. Three examples are examined, both for finite and infinite dimensional groups associated with geometric and signature variation in image understanding (1,2) and anatomical shape representation (3). 1. U. Grenander, M. I. Miller and A. Srivastava, "Hilbert-Schmidt Lower Bounds for Estimators on Matrix Lie Groups for ATR,'' IEEE Trans. on Pattern Analysis and Machine Intelligence, November 1998. 2. E. Shusterman, M. I. Miller and B. Rimoldi, "Rate-Distortion Theoretic Design of Dictionaries for Object Recognition,'' Research Monograph Center for Imaging Sciences, 1997. 3. U. Grenander and M. I. Miller, "Computational Anatomy: An Emerging Discipline,'' Quarterly of Applied Mathematics, pp. 617-694, 1998. (*) This work was supported by Grant ARO DAAH-04-95-1-0494, ONR-MURI N00014-98-1-0606.  

Abstract

It is becoming increasingly easy to acquire and maintain large audio collections, and audio search techniques that incorporate speech recognition technology are evolving rapidly. Less is known, however, about the strategies that users will adopt when searching large audio collections. Information retrieval is a synergistic process in which the user and the system seek to exploit each other's strengths and cover each other's weaknesses, and audio retrieval shifts this balance in several ways when compared with text retrieval. In this talk I will describe some important differences between retrieval of speech and written text and explore the design space in which a new balance might be found. I will then describe the VoiceGraph project at the University of Maryland in which we are using iterative prototyping to design audio retrieval interfaces. I'll conclude with a few remarks on how what we learn in the VoiceGraph project might inform future work on component technologies such as speech recognition, speaker identification, and topic boundary detection.

Abstract

We seek to gain better understanding of sensory information processing in physical systems both natural and engineered. For an information processing system the statistics of the input, the details of the algorithm, and the task requirements determine the minimum information transmission rate, R (bits/sec). The properties of the physical substrate, such as bandwidth, noise and constraints on the signal value, determine the channel capacity, C (bits/sec). These physical properties must be such that C > R for reliable performance. Furthermore, any implementation involves tradeoffs among costs such as power, speed, accuracy, and area. We seek better understanding of the compromise between performance and cost. To date there exist several measurements of the information transmission rates for spiking and non-spiking neurons. These measurements must be supported by the physical properties of the communication channel, and we explore this relationship for one of these systems, the blowfly retina. We construct a communication channel model that incorporates all physical transformations from photons at the photoreceptor to the membrane voltage of the large monopolar cell in the laminar layer. In this talk I will begin with a brief historical review of previous work that employs information theoretic ideas to analyze neural information processing. I will then describe the components of the early vision system in the blowfly retina. From biophysical data available in the literature, we determine bandwidth limitations and noise contributions at the different stages and calculate the Shannon capacity of the system. We compare our model with empirical information capacities derived from measurements on the system. I will conclude my talk by briefly discussing future work aimed at determining the energy efficiency of the system as given by bit-energy, the ratio of information rate to power dissipated. References: P. Abshire and A.G. Andreou, "Relating Information Capacity to a Biophysical Model of the Blowfly Retina", Electrical and Computer Engineering, Technical Report 13-1998. A.G. Andreou and P.M. Furth, "An Information Theoretic Framework for Comparing the Bit-Energy of Signal Representations at the Circuit Level," Chapter 17, Low-Voltage/Low-Power Integrated Circuits and Systems, edited by Edgar Sanchez-Sinencio and Andreas G. Andreou, IEEE Press, 1998. Work supported by a DARPA/ONR Multidisciplinary University Research Initiative with Boston University on Automated Sensing and Vision Systems N00014-95-1-0409.  

Abstract

The late 15th century Voynich manuscript is written in an unknown cipher script that has resisted all attempts at reading for 80 years. But maybe it's really not a cipher at all but a hoax, a madman's scribbles, written in an invented language, or a written equivalent of glossolallia. The Book of Soyga, studied by John Dee (1527-1608), is a 16th century magic treatise with a hidden mathematical surprise. And Book 3 of the 1500 Steganographia by Johannes Trithemius (1462-1516) contains recently discovered hidden cipher messages. Jim Reeds will tell the tale of his adventures with these puzzles.  

Abstract

Plesionyms, or near-synonyms, are words, that, within or across languages, are almost synonyms---but not quite. Some examples: "forest", "woods", German "Wald"; "fib", "lie", "misrepresentation". Near-synonyms may differ in one or more of the following: connotation, emphasis on subcomponents, implicature, denotation, speaker's expressed attitude, register, and structural or selectional requirements. In all but the last two of these, the distinction between two near-synonyms is at least in part conceptual. It is necessary to represent lexical meaning finely enough that distinctions between near-synonyms can adequately be taken into account in such tasks as lexical choice in machine translation and mono- and multilingual text generation. This is the basis for an alternative to conventional models of the relationship between words and concepts: a coarse-grained hierarchy in which clusters of near-synonyms are distinguished by explicit differentiae. This model is implemented in a system for lexical choice that is envisioned as a component of high-quality machine translation.  

Abstract

Mass deployment of computing technology depends upon ease-of-use and natural human/machine interfaces. Communication methods based upon the sensory modes preferred by the human--sight, sound and touch--are consequently being developed. While as yet primitive, multimodal interfaces depend centrally upon conversational interaction. And, integration of speech with visual and tactile capabilities centers on fusing simultaneous sensory inputs (which are often redundant, ambiguous or contradictory) to achieve reliable interpretation and action. This report describes one in-progress research effort in multimodal interfaces that employs simultaneous eye tracking, visual gesture, hands-free sound capture, speech recognition, text-to-speech synthesis, tactile force-feedback and manual gesture.

Abstract

Repetition is very common. Adaptive language models were introduced to account for the fact that words (and their variant forms) tend to appear in bursts. We will show that this is especially true for words with a lot of content such as proper nouns, technical terminology and good keywords for information retrieval. A proper noun like ``Kennedy'' is more likely to be repeated in a Brown Corpus document than a common noun like ``showed,'' even though both words are about equally frequent. We find that words (and ngrams) with more content tend to be more bursty than words (and ngrams) with less content, all other things being equal. Measures borrowed from Information Retrieval, term frequency and document frequency, will be used to predict both the average frequency and the variance (burstiness) of a word. The literature on adaptive language models has studied the first moment in considerable detail, but has tended to ignore the second moment.

Abstract

This talk will describe Bayesian networks, and place them in the context of automatic speech recognition. The Bayesian network formalism has both representational and algorithmic components, and the talk will touch on each. Representationally, the networks provide a graphical way of factoring a joint probability distribution. The nodes in a Bayes net graph represent random variables, whose values can be either known or unknown. The arcs in the graph factor the joint distribution into a product of localized conditional probabilities, each of which involves only a few variables. The conditional probabilities can be represented with either tables, Gaussians, or any other convenient function. Algorithmically, there are elegant and efficient procedures for computing marginal distributions over the values of the hidden variables, and for finding the likeliest assignment of values. In ASR, these lead directly to the computation of state-occupancy probabilities and Viterbi decodings. The key feature of Bayesian networks is that the algorithms are parameterized on the graph structure and the representation of conditional probabilities. This makes it very easy to explore a variety of probabilistic models with a minimum of code-writing. In addition to describing the basic algorithms, the talk will relate Bayesian networks to the HMMs currently in use in ASR, and show how they provide a simple method for extending HMMs to model phenomena such as rate-of-speech and articulatory motion.  

Abstract

Researchers of natural language have repeatedly acknowledged that coherent texts are not just simple sequences of sentences. Rather, they are complex artifacts whose semantic units are connected by rhetorical, logical, argumentative, and cohesive relations. I present research in theoretical, empirical, and applied computational linguistics that aims at uncovering the constraints that characterize the abstract structure of well-formed texts, and at producing algorithms for the automatic derivation of these structures. I show how automatically constructed discourse structures are exploited in a text summarization system and discuss other text processing open problems that can be properly addressed in a discourse-based framework

Abstract

Telecommunication in recent years is experiencing a revolution in progress. The advant of Internet and its exponentially rapid growth has triggered an entirely new thinking in terms of the means to achieve communication. The old paradigm of telephony has been shifted or broadened to video conferencing, distance learning, and remote collaboration and access to multimedia databases, offering both flexibility and richness in media content, management and human-machine interface. While significant contributions to this bright new world of telecommunciation come from infrastructure technologies such as switches, routers and optical networks, equally important, if not more so, is the advance in various signal processing areas which form the drivers of the new paradigm. These include high quality acoustics, audio processing & distribution, speech recognition, biometric authentication and signal synthesis. Systems that integrate these technologies will further bring us mobility, convenience and functionality that will drive the need of communication bandwidth and quality in the future. In this talk, I'll summarize advances made during the last few years in these signal processing technical areas and highlight challenges ahead of us that need to be overcome to realize the ultimate dream of a multimedia era with natural human-machine interface.  

Abstract

The Information Extraction (IE) task has driven a substantial body of natural language processing research during the past decade. We begin by presenting an overview of IE, including a description of the class of problems IE addresses, the manner in which IE systems are generally evaluated, and a typical IE system architecture. After a brief overview of previous work in applying machine learning techniques to IE problems, we describe some ambitious (and largely unsuccessful) attempts to learn discourse interpretation strategies within such a system. The results of this work led to a re-examination of the evaluation metrics used to drive the learning process, which has revealed some unforeseen attributes that should be avoided in future evaluation schemes.

Abstract

Bayesian belief networks (BBNs) are a class of joint distribution functions that have a graphical representation. Inference with BBN's is performed by propagating probabilities throughout the graph via repeated application of Bayes' rule. Both inference and approximate inference in general BBNs is known to be NP-hard and thus BBNs cannot be applied to large-scale systems without the development of efficient approximate algorithms. The similarity between certain parameterized BBNs (e.g. sigmoid belief networks) and complex physical systems (e.g. the Ising model of ferromagnetism) recently prompted the successful application variational methods to the problem of approximate inference with parameterized BBNs. After a brief overview, this talk will focus on two novel approximate inference algorithms for a class of parameterized belief networks. The algorithms are a consequence of the application of saddle-point methods from statistical physics. The first algorithm yields a previously unknown and easy to calculate upper bound on posterior probabilities. The second algorithm is a Gaussian approximation that is significantly more precise than upper- and lower-bound techniques and takes into account correlations between random variables. If time permits, the proposed application of these algorithms to the problem of rapid microorganism identification via database search will be discussed.

Abstract

The ratio spectrum is a novel spectral representation that has shown promise for improved speech compression, coding, feature extraction and recognition. In effect, the ratio spectrum combines the standard front-end filter bank with a feature extraction process to produce a model that requires dramatically less hardware (or software). Alternatively, the model can be viewed as a small set of constant-Q filters whose center frequencies adapt to locations of high signal energy. The resulting feature vectors are shown to outperform several competing techniques for phoneme recognition (e.g. LPC and cepstrum). We also have implemented speech and audio coding using the ratio spectrum and standard spectrum inversion techniques. Finally, results from fabricated CMOS analog VLSI circuits illustrate a hardware efficient method to sample the ratio spectrum, paving the way for ultra low-power front-end speech processing and feature extraction.

Abstract

What is the neural representation of a speech code as it evolves in time? How do listeners integrate temporally distributed phonemic information into coherent representations of syllables and words? How does the brain extract invariant properties of variable-rate speech? This talk will describe an emerging neural model that suggests answers to these questions, while quantitatively simulating challenging data about speech and word recognition. In this model, rate-dependent category boundaries emerge from feedback interactions between a working memory for short-term storage of phonetic items and a list categorization network for grouping sequences of items. The conscious speech and word recognition code is suggested to be a resonant wave. Such a wave emerges when sequential activation and storage of phonemic items in working memory provides bottom-up input to unitized representations, or list chunks, that group together sequences of items of variable length. The list chunks compete with each other as they dynamically integrate this bottom-up information. The winning groupings feed back to provide top-down support to their phonemic items. These top-down expectations amplify and focus attention on consistent working memory items, while suppressing inconsistent working memory items. Feedback establishes a resonance which temporarily boosts the activation levels of selected items and chunks, thereby creating an emergent conscious percept. Because the resonance evolves more slowly than working memory activation, it can be influenced by information presented after relatively long intervening silence intervals. Variations in the durations of speech sounds and silent pauses can hereby produce different perceived groupings of words, and future sounds can influence how we hear past sounds. Preprocessing of acoustic signals into parallel auditory streams that respond preferentially to transient and sustained properties of the acoustic signal before being stored in parallel working memories, together with cross-stream automatic gain control, can help to explain how an invariant speech representation can emerge from variable-rate speech. References: Boardman, I., Grossberg, S., Myers, C., and Cohen, M. (1998). Neural dynamics of perceptual order and context effects for variable-rate speech syllables. Perception & Psychophysics, in press. Grossberg, S., Boardman, I., and Cohen, M. (1997). Neural dynamics of variable-rate speech categorization. J. Exptal. Psychol.: Human Percept. & Perform., 23, 481-503. Grossberg, S. and Myers, C. (1999). The resonant dynamics of speech perception: Interword integration and duration-dependent backward effects. Psychological Review, in press.

Abstract

Speech perception involves the parallel processing of a wide range of information. Audiovisual speech perception is the best example of this parallel processing with the visual modality providing complementary information to the auditory channel. Under noisy conditions visible speech movements supplement the auditory signal intelligibility is increased Even under very good listening conditions visible speech can influence normal auditory perception. The McGurk effect is an auditory-visual speech illusion in which the perception of a speech sound is modified by contradictory visual information. In this presentation I will present data from speech in noise experiments and studies of the McGurk effect. My goal in this work is to characterize the visual information processing involved in audiovisual speech perception. As part of this research program we have developed facial animation techniques that allows us to control the visual input. A significant weakness in the existing audiovisual speech literature is the failure to control the visual stimulus characteristics. I will summarize our modeling efforts and demonstrate the benefits of this approach.