What Is ASR  |  Tools  |  Research  |  Past Projects  |  Current and Ongoing Projects  |  People  |  Publications  |  Education  |  Links  |  Sponsors

• The Tucker-Maxon project
At the Tucker Maxon Oral School in Portland, Oregon, profoundly deaf children whose hearing is enhanced through amplification or electrical stimulation of the cochlea are improving their listening and speech production skills by conversing with Baldi, an animated conversational agent. One component of conversing with Baldi is automatic recognition of a deaf child's speech. This is an extremely challenging area for ASR, and involves the development of children's speech recognizers and properly rejecting incorrectly-pronounced words. The Tucker-Maxon project was supported by NSF grant CDA-9726363.
• The Spoltech project
The Spoltech project was a collaboration with our partners in Porto Alegre, Brazil, Caxias do Sul, Brazil, and Boulder, Colorado, for the development of a Brazilian Portuguese version of the CSLU Toolkit. The Spoltech project was supported by NSF grant IIS-9970061 and a grant from CNPq.
• Automatic phonetic alignment
The topic of time-aligning a given sequence of words or phonemes with a known speech signal is important for several areas of speech research, and is also strongly linked with the topic of speech recognition. Time-aligned phonemes are useful in developing speech recognizers, speech synthesis systems, animated faces, and in studies on coarticulation and prosody. Our approaches to this problem include detecting and utilizing instants of phoneme-related change in different energy bands of the speech signal (notably the work done by Jan van Santen), extracting and integrating acoustic-phonetic information (such as voicing and burst information) into the HMM/ANN framework, and classifying and integrating phonetic transition information into the HMM framework. Although our results are currently state-of-the-art (with 92.6% agreement within 20 msec on the TIMIT corpus), we have plans for further reducing the error rate. Our work on automatic phonetic alignment has been supported by NSF grants GRT 9354959 and CDA-9726363.
• ROAR
The framework for most ASR systems, the Hidden Markov Model (HMM), has been in place for approximately 20 years, and most improvements in ASR performance operate within this model. However, it is possible that, despite the HMM framework being well-understood, the inaccurate assumptions made when using HMMs for processing speech leads to shortcomings in performance. The goal of this work was to challenge the current architecture and dogma in core ASR technologies by proposing and prototyping a high-risk, high-yield alternative to standard HMMs.

The proposed alternative ASR system performs segment-based context-dependent phonetic classification over a wide range of starting and ending frames, keeps only the phonemes that have sufficient likelihood for creation of a lattice of possible phonemes, and then employs a dynamic-programming search through this phoneme lattice to match the identified phonemes with the possible words. The use of segment-based classification is similar in some respects to the SUMMIT system developed at MIT; in contrast with SUMMIT, an explicit segmentation stage prior to phoneme classification is not expected. This combination of frame-based and segment-based approaches is called an "acoustic-fragment" based approach. This project, part of the ROAR series of projects, was sponsored by DARPA.

• Making Dsyarthric Speech Intelligible
This project [joint with Melanie Fried-Oken at the Child Development and Rehabilitation Center at the Oregon Health and Science University] will develop new algorithms that will enable dysarthric individuals to be more easily understood. Currently available devices are essentially spectral filters and amplifiers that enhance certain parts of the spectrum. While these can help certain types of dysarthria, many dysarthric persons suffer from speech problems that require forms of speech modification that are much more profound and complex such as: irregular sub-glottal pressure, resulting in loudness bursts that can be difficult to adjust to; absence, or poor control, of voicing; systematic mispronunciation of certain phoneme groups, resulting in certain sounds becoming indistinguishable or unrecognizable; variable mispronunciation; and poor prosody (pitch control, timing, and loudness). For these difficult problems, new approaches are needed that do not merely filter the speech signal but analyze it at acoustic, articulatory, phonetic, and linguistic levels.

• Vietnamese Speech Recognition
The CSLU Toolkit has been used for Vietnamese recognition. A continuous digit recogntion system for Vietnamese has been developed with the National Centre for Natural Science and Technololgy of Vietnam, Institute of Information Technology. A Vietnamese digit corpus has been built for this recognizer. The corpus consists of 442 sentences (2345 words), which were extracted from two of CSLU's corpora, "22 Language v1.2" and "Multi-Language Telephone Speech v1.2". The best result achieved is 97.46% word accuracy and 90.41% sentence correct. For more detail, refer to http://duc.netfirms.com/

• Collaboration with Piero Cosi at C.N.R.'s Institute of Phonetics and Dialectology (IFD)
Our collaboration with Piero Cosi dates back several years, and we have been collaborating both electronically and, with help from C.N.R. and I.F.D., face-to-face. We have investigated the relative performance of standard HMMs and HMM/ANN hybrid systems, techniques for improving HMM/ANN performance, especially for connected digit recognition, and the development of Italian continuous-digit and general-purpose recognizers. More recently, Piero has been researching and developing Italian text-to-speech and facial animation.

CSE540:    Neural Network Algorithms and Architectures (Leen)
CSE544:    Introduction toProbability and Statistical Inference (Yang)
CSE547:    Statistical Pattern Recognition (Leen)
CSE548:    Modern Applied Statistics (Yang)
CSE550:    Spoken Language Systems (Heeman)
CSE551:    Structure of Spoken Language (Staff)
CSE552:    Hidden Markov Models for Speech Recognition (Hosom)
CSE58X:   Statistical Natural Language Processing (Heeman)
CSE561:    Dialogue (Cohen)
CSE562:    Natural Language Processing (Staff)
ECE 540:  Auditory and Visual Processing by Human and Machine (Hermansky, Pavel)
ECE 541:  Speech Processing (Hermansky)
ECE 545:  Speech Systems (Hermansky)
ECE 547:  Signals for Multimedia Engineering (Hermansky)

CSLU (Center for Spoken Language Understanding)
CSLR (Center for Speech and Language Research)
CMU's Speech Group
ICSI (International Computer Science Institute)
ISIP (Institute for Signal and Information Processing)
MIT Spoken Language Systems Group
PSL (the Perceptual Sciences Laboratory at UCSC)
Bell Labs: Statistical Models for Speech Recognition
Tony Robinson / Cambridge
Tlatoa
LDC (Linguistic Data Consortium)

The National Science Foundation (NSF)

Defense Advanced Research Projects Agency (DARPA)

Istituto di Scienze e Tecnologie della Cognizione Sezione di Padova "Fonetica e Dialettologia" Consiglio Nazionale delle Ricerche (ISTC-SPFD, CNR, Italy)

Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq, National Council for Scientific and Technological Development of Brazil)

Last updated in November of 2003.
For questions or comments about this page, contact John-Paul Hosom.