自然手语动作序列生成的研究

文章探讨了如何让在手语新闻播报中的卡通人按照自然手语的语法规则而非正常人的语法规则来打手语。首先整理了现代汉语自然手语的规则并将其形式化,并建立了正常汉语到汉语自然手语转换的形式规则库;从而实现了现代汉语文本到相应的自然手语的手语动作序列的自动生成。最后将其嵌入到通过手语合成技术和卡通动画的手语新闻播报系统中,使其在线输出的是符合聋人习惯的自然手语。     

Interactive editing in French Sign Language dedicated to virtual signers: requirements and challenges

Signing avatars are increasingly used as an interface for communication to the deaf community. In recent years, an emerging approach uses captured data to edit and generate sign language (SL) gestures. Thanks to motion editing operations (e.g., concatenation, mixing), this method offers the possibility to compose new utterances, thus facilitating the enrichment of the original corpus, enhancing the natural look of the animation, and promoting the avatar’s acceptability. However, designing such an editing system raises many questions. In particular, manipulating existing movements does not guarantee the semantic consistency of the reconstructed actions. A solution is to insert the human operator in a loop for constructing new utterances and to incorporate within the utterance’s structure constraints that are derived from linguistic patterns. This article discusses the main requirements for the whole pipeline design of interactive virtual signers, including: (1) the creation of corpora, (2) the needed resources for motion recording, (3) the annotation process as the heart of the SL editing process, (4) the building, indexing, and querying of a motion database, (5) the virtual avatar animation by editing and composing motion segments, and (6) the conception of a dedicated user interface according to user’ knowledge and abilities. Each step is illustrated by the authors’ recent work and results from the project Sign3D, i.e., an editing system of French Sign Language (LSF) content (http://sign3d.websourd.org/sltat).     

Chinese Sign Language animation generation considering context

Sign language (SL) is a kind of natural language for the deaf. Chinese Sign Language (CSL) synthesis aims to translate text into virtual human animation, which makes information and service accessible to the deaf. Generally, sign language animation based on key frames is realized by concatenating sign words captured independently. That means a sign language word has the same pattern in diverse context, which is different from realistic sign language expression. This paper studies the effect of context on manual gesture and non-manual gesture, and presents a method for generating stylized manual gesture and non-manual gesture according to the context. Experimental results show that synthesized sign language animation considering context based on the proposed method is more accurate and intelligible than that irrespective of context.     

Collecting and evaluating the CUNY ASL corpus for research on American Sign Language animation

While there is great potential for sign language animation generation software to improve the accessibility of information for deaf individuals with low written-language literacy, the understandability of current sign language animation systems is limited. Data-driven methodologies using annotated sign language corpora encoding detailed human movement have enabled some researchers to address several key linguistic challenges in ASL generation. This article motivates and describes our current research on collecting a motion-capture corpus of American Sign Language (ASL). As an evaluation of our motion-capture configuration, calibration, and recording protocol, we have conducted several rounds of evaluation studies with native ASL signers, and we have made use of our collected data to synthesize novel animations of ASL, which have also been evaluated in experimental studies with native signers.     

面向广电节目的虚拟人手语合成显示平台研究

基于中国手语合成技术的虚拟人手语视频显示平台技术是一个全新的课题.为了满足广电新闻节目对手势运动流畅性要求,实现了一种上下文相关的手势运动平滑算法,该方法能够充分利用前后两帧的差异来实现手势运动的平滑过渡,其视觉感观效果较传统插值算法更加平滑自然;同时提出了一种基于统计和规则相结合的手势运动重定向算法,在统计方法的基础上针对不同骨架大小以及运动特性进行规则约束,使得标准模型手势运动数据应用到新模型上而不失其准确性;最后,通过扩展基本手语词表达形式并基于alpha融合技术实现了面向广电新闻节目的虚拟人手语合成显示平台并取得很好的结果.     

Temporal Contexts for Discourse Representation: An Extension of the Conceptual Graph Approach

A discourse is composed of a sequence of sentences that must be interpreted with respect to the context in which they are uttered and to the actions that produce them: locutors' speech acts. The analysis of discourse content must be based on a pragmatic approach to the study of language in use. Some of the most obvious linguistic elements that require contextual information for their representation are deictic forms such as here, now, I, you, this , and verb tenses.Several authors have recognized a need for introducing contextual structures in knowledge representation models such as semantic networks. Sowa's Conceptual Graph Theory is a powerful approach to conceptually represent knowledge contained in discourses. However, it must be extended in order to represent several semantic and pragmatic mechanisms related to the expression of time in natural language. In this paper we present such an extension as a framework for modeling temporal knowledge in discourses integrating several features borrowed from speech act theory.First, we introduce the notions of time interval, temporal object, temporal situation, and temporal relation. Then, we discuss the importance of explicitly introducing the concept of time coordinate system in a discourse representation and we present different kinds of temporal contexts: narrator's perspective, agent's perspective and temporal localization. We show how this conceptual framework can be used to represent various referential mechanisms in discourse such as anaphoras, indexicals, direct and indirect styles. We also discuss how to model several linguistic phenomena such as speech act characteristics and the specification of performative and attitude utterances. Finally, we briefly discuss how verb tenses can be determined in a discourse on the basis of this temporal approach.     

A real-time interactive nonverbal communication system through semantic feature extraction as an interlingua

There has been a growing interest in the use of networked virtual environment (NVE) technology to implement telepresence that allows participants to interact with each other in shared cyberspace. In addition, nonverbal language has attracted increased attention because of its association with more natural human communication, and especially sign languages play an important role for the hearing impaired. This paper proposes a novel real-time nonverbal communication system by introducing an artificial intelligence method into the NVE. We extract semantic information as an interlingua from the input text through natural language processing, and then transmit this semantic feature extraction (SFE) to the three-dimensional (3-D) articulated humanoid models prepared for each client in remote locations. Once the SFE is received, the virtual human is animated by the synthesized SFE. Experiments with Japanese and Chinese sign languages show this system makes the real-time animation of avatars available for the participants when chatting with each other. The communication is more natural since it is not just based on text or predefined gesture icons. This proposed system is suitable for sign language distance training as well.     

Computer animated visual translation from natural language to sign language

Computer animation and visualization can facilitate communication between the hearing impaired and those with normal speaking capabilities. This paper presents a model of a system that is capable of translating text from a natural language into animated sign language. Techniques have been developed to analyse language and transform it into sign language in a systematic way. A hand motion coding method as applied to the hand motion representation, and control has also been investigated. Two translation examples are also given to demonstrate the practicality of the system.     

KAZOO: a sign language generation platform based on production rules

This paper describes KAZOO, a web application for sign language (SL) generation using a virtual signer. Firstly, it explains the motivation to this project, which is grounded on an approach designed solely from SL corpus analysis and modelling. Then, various projects conducted in the past few years on linguistic modelling and 3D animation are presented. The platform’s architecture integrates parts of this work and new pieces of software allowing control and linking of all these components. This is an ongoing project, though the current version offers the possibility to animate a virtual signer automatically and synthesize the contents using an abstract representation, the authors’ own linguistic model AZee, providing a means of validating this model.     

A human-machine cooperative system for generating sign language animation using thermal image

We propose a new approach aimed at sign language animation by skin region detection on an infrared image. To generate several kinds of animations expressing personality and/or emotion appropriately, conventional systems require many manual operations. However, a promising way to realize a lower workload is to manually refine an animation made automatically with a dynamic image of real motion. In the proposed method, a 3D CG model corresponding to a characteristic posture in sign language is made automatically by pattern recognition on a thermal image, and then a person’s hand in the CG model is set. The hand part is made manually beforehand. If necessary, the model can be replaced manually by a more appropriate model corresponding to training key frames and/or the model can be refined manually. In our experiments, a person experienced in using sign language recognized the Japanese sign language of 71 words expressed as animation with 88.3% accuracy, and three persons experienced in using sign language also recognized the sign language animation representing three emotions (neutral, happy and angry) with 88.9% accuracy. This work was presented in part at the 13th International Symposium on Artificial Life and Robotics, Oita, Japan, January 31–February 2, 2008     

High-level interpretation in virtual environments

As virtual environments applications become more complex, there is a need to interpret user interaction in terms of high-level concepts. In this paper, the relations between conceptual representations of actions and their physical simulation in virtual worlds is investigated. A model inspired fromnatural language processing (NLP) research in the linguistic interpretation of dynamic scenes has been developed. The experiments are based on real-time animation software, which has been enhanced with a symbolic information processing layer, originally developed for NLP-based animation. We report the implementation of a high-level interpretation module that is able to recognize complex actions fromlow-level physical events in the virtual world and discuss its performance as well as directions for further developments.     

基于状态机的手语动画自动生成技术

目前,手语的资源主要是图片、视频和动画。由于这些资源相对比较固定,无法动态扩展,不能满足手语动画自动生成的需求。论文将状态机应用于手语动画生成中,设计了基于状态机的手语动画自动生成技术算法,并将该算法应用到公交报站手语自动生成系统中。实验结果表明,生成的手语动作流畅,准确率达97%,具有良好的市场应用前景。     

Dynamic–static unsupervised sequentiality,statistical subunits and lexicon for sign language recognition

We introduce a new computational phonetic modeling framework for sign language (SL) recognition. This is based on dynamic–static statistical subunits and provides sequentiality in an unsupervised manner, without prior linguistic information. Subunit “sequentiality” refers to the decomposition of signs into two types of parts, varying and non-varying, that are sequentially stacked across time. Our approach is inspired by the Movement–Hold SL linguistic model that refers to such sequences. First, we segment signs into intra-sign primitives, and classify each segment as dynamic or static, i.e., movements and non-movements. These segments are then clustered appropriately to construct a set of dynamic and static subunits. The dynamic/static discrimination allows us employing different visual features for clustering the dynamic or static segments. Sequences of the generated subunits are used as sign pronunciations in a data-driven lexicon. Based on this lexicon and the corresponding segmentation, each subunit is statistically represented and trained on multimodal sign data as a hidden Markov model. In the proposed approach, dynamic/static sequentiality is incorporated in an unsupervised manner. Further, handshape information is integrated in a parallel hidden Markov modeling scheme. The novel sign language modeling scheme is evaluated in recognition experiments on data from three corpora and two sign languages: Boston University American SL which is employed pre-segmented at the sign-level, Greek SL Lemmas, and American SL Large Vocabulary Dictionary, including both signer dependent and unseen signers' testing. Results show consistent improvements when compared with other approaches, demonstrating the importance of dynamic/static structure in sub-sign phonetic modeling.     

The Translator’s Extended Mind

The rapid development of natural language processing in the last three decades has drastically changed the way professional translators do their work. Nowadays most of them use computer-assisted translation (CAT) or translation memory (TM) tools whose evolution has been overshadowed by the much more sensational development of machine translation (MT) systems, with which TM tools are sometimes confused. These two language technologies now interact in mutually enhancing ways, and their increasing role in human translation has become a subject of behavioral studies. Philosophers and linguists, however, have been slow in coming to grips with these important developments. The present paper seeks to fill in this lacuna. I focus on the semantic aspects of the highly distributed human–computer interaction in the CAT process which presents an interesting case of an extended cognitive system involving a human translator, a TM tool, an MT engine, and sometimes other human translators or editors. Considered as a whole, such a system is engaged in representing the linguistic meaning of the source document in the target language. But the roles played by its various components, natural as well as artificial, are far from trivial, and the division of linguistic labor between them throws new light on the familiar notions that were initially inspired by rather different phenomena in the philosophy of language, mind, and cognitive science.