多模式汉语连续语音识别中视觉特征的提取和应用

本文对在汉语多模式汉语语音识别系统中利用视觉特征进行了研究,给出了基于多流隐马尔科夫模型(Multi-stream HMM, MSHMM)的听视觉融合方案,并对有关视觉特征的两项关键技术:嘴唇定位和视觉特征提取进行了详细讨论。首先,我们研究了基于模板匹配的嘴唇跟踪方法;然后研究了基于线性变换的低级视觉特征,并与基于动态形状模型的特征作了比较;实验结果表明,引入视觉信息后无噪环境下语音识别声学层首选错误率相对下降36.09%,在噪声环境下的鲁棒性也有明显提高。

To Extract and Apply Visual Features in Mandarin Multimodal Continuous Speech Recognition

In this paper, we investigate on the using of visual feature in Mandarin multimodal speech recognition. The audio visual fusion strategy based on multi stream hidden Markov model is presented. Then key technologies about visual feature, including lip location and visual feature extraction, are discussed. Firstly, we research on the lip location algorithm based on model matching and the low Subsequently, the low level visual feature based on linear transform is investigated and compared to the high level visual feature based on active shape models. It is shown by experiments that the word error rate of the first candidate of acoustic level is reduced by 36 09% relatively with visual feature used, compared to audio speech recognition system. It is also demonstrated from more experiments that our audio visual system provides significant robustness enhancement in noise environment.