Computer animated visual translation from natural language to sign language

Computer animation and visualization can facilitate communication between the hearing impaired and those with normal speaking capabilities. This paper presents a model of a system that is capable of translating text from a natural language into animated sign language. Techniques have been developed to analyse language and transform it into sign language in a systematic way. A hand motion coding method as applied to the hand motion representation, and control has also been investigated. Two translation examples are also given to demonstrate the practicality of the system.     

Educational resources and implementation of a Greek sign language synthesis architecture

In this paper, we present how creation and dynamic synthesis of linguistic resources of Greek Sign Language (GSL) may serve to support development and provide content to an educational multitask platform for the teaching of GSL in early elementary school classes. The presented system utilizes standard virtual character (VC) animation technologies for the synthesis of sign sequences/streams, exploiting digital linguistic resources of both lexicon and grammar of GSL. Input to the system is written Greek text, which is transformed into GSL and animated on screen. To achieve this, a syntactic parser decodes the structural patterns of written Greek and matches them into equivalent patterns of GSL, which are then signed by a VC. The adopted notation system for the representation of GSL phonology incorporated in the system’s lexical knowledge database, is Hamburg Notation System (HamNoSys). For the implementation of the virtual signer tool, the definition of the VC follows the h-anim standard and is implemented in a web browser using a standard VRML plug-in.     

Motion synthesis and editing for the generation of new sign language content

Machine Translation - Existing work on the animation of signing avatars often relies on pure procedural techniques or on the playback of Motion Capture (MoCap) data. While the first solution...     

Computation of the structured stability radius via matrix sign function

Robustness of perturbed state space models of the form is considered, where B, C are given matrices, A is an asymptotically stable matrix and D is the unknown perturbation matrix. An efficient algorithm to compute the complex structured stability radius, which is based on the properties of the matrix sign function, is presented. A comparison with previous algorithms shows the efficiency of the new algorithm     

The application of structured learning in natural language processing

We propose a structured learning approach, max-margin structure (MMS), which is targeted at natural language processing (NLP) tasks. The architecture of our approach is shown to capture structural aspects of the problem domains, leading to demonstrable performance improvements on two NLP tasks: part-of-speech tagging and statistical machine translation (SMT). We present a perceptron-based online learning algorithm to train the model and demonstrate desirable computational scaling behavior over traditional optimisation methods.     

Performance analysis of Otsu thresholding for sign language segmentation

Multimedia Tools and Applications - Sign language recognition system generally consists of three main processes, which are segmentation, modelling, and classification. Image segmentation plays a...     

Synthesis of sign language co-articulation based on key frames

Multimedia Tools and Applications - Co-articulation is a language phenomenon. In sign language (SL), it takes the form of impact among adjacent signs, which results in variations of signs from...     

Optimized H.264 compression of sign language video

In TV broadcasting deaf people are not able to get information from the audio content. In public television, some programs may be accompanied by a sign language interpreter as a part of the broadcasted signal. As a supplementary service, it would enable more programs to be accessible with a sign language interpreter to assist in comprehension. To be able to transmit such data flow separately, we define the parameters of compression of sign language interpreter image to ensure intelligibility and quality while maintaining low bitrate. This paper deals with specific video compression of Czech sign language interpreter based on regions of interest implemented to the ×264 open source library. The results of this approach are verified in subjective tests with the deaf and hearing evaluators. The experiments examine the intelligibility of sign language expressions containing minimal pairs for different levels of image compression and also evaluate the subjective quality of the final image.     

Improving realism in automated fingerspelling of American sign language

Machine Translation - Fingerspelling is a process of communicating letters of a spoken language alphabet using a person’s hand or hands. Portraying animations of fingerspelling has proved...     

手语识别中基于HMM的区分性训练方法

传统的隐马尔科夫模型（HMM）的训练方法基于统计概率的最大似然准则（MLE），在训练样本数目足够大的情况下，这种方法在理论上可以得到最优的结果．在手语识别研究中，采集足够大的训练样本十分困难．区分性训练可以很好地弥补由于训练样本的缺乏以及手语模型之间的近似而造成的识别系统的缺陷．最大交互信息准则（MMIE）作为区分性训练准则的一种已经被广泛的应用于语音识别领域．文中通过合理的构建手语识别中的竞争模型和易混集。提出了MMIE准则的改进形式，并将其应用于特定人与非特定人手语识别．实验证明，使用改进的MMIE准则对识别系统性能有很大的提高．     

Increasing adaptability of a speech into sign language translation system

This paper describes a new version of a speech into sign language translation system with new tools and characteristics for increasing its adaptability to a new task or a new semantic domain. This system is made up of a speech recognizer (for decoding the spoken utterance into a word sequence), a natural language translator (for converting a word sequence into a sequence of signs belonging to the sign language), and a 3D avatar animation module (for playing back the signs). In order to increase the system adaptability, this paper presents new improvements in all the three main modules for generating automatically the task dependent information from a parallel corpus: automatic generation of Spanish variants when generating the vocabulary and language model for the speech recogniser, an acoustic adaptation module for the speech recogniser, data-oriented language and translation models for the machine translator and a list of signs to design. The avatar animation module includes a new editor for rapidly design of the required signs. These developments have been necessary to reduce the effort when adapting a Spanish into Spanish sign language (LSE: Lengua de Signos Española) translation system to a new domain. The whole translation presents a SER (Sign Error Rate) lower than 10% and a BLEU higher than 90% while the effort for adapting the system to a new domain has been reduced more than 50%.     

Recognition of words expressed by sign language using thermal-image processing

A basic method for recognizing sign language has been developed with thermal-image processing. In this method, a pair of traces of two hands, which corresponds to a word in Japanese sign language, is input into a computer as sequential coordinates with thermal-image processing. Then the word is recognized by finding the most similar pair of traces in the database to that expressing the word. Forty-seven Japanese words in sign language were recognized with 71% accuracy. This work was presented in part at the 11th International Symposium on Artificial Life and Robotics, Oita, Japan, January 23–25, 2006     

Algebraic-grammatical specifications and synthesis of structured program schemas

The paper proposes hierarchical specifications of classes of algorithms and programs (both sequential and parallel) in algebras of hyperschemas (AHS), based on Glushkov's systems of algorithmic algebras. Derivation-control algorithms for various grammatical models are presented in AHS.Translated from Kibernetika, No. 6, pp. 5–16, November–December, 1989.     

Analysis, preparation, and optimization of statistical sign language machine translation

Sign languages represent an interesting niche for statistical machine translation that is typically hampered by the scarceness of suitable data, and most papers in this area apply only a few, well-known techniques that are not adapted to small-sized corpora. In this article, we analyze existing data collections and emphasize their quality and usability for statistical machine translation. We also offer findings in the proper preprocessing of a sign language corpus, by introducing sentence end markers, splitting compound words and handling parallel communication channels. Then, we focus on optimization procedures that are tailored to scarce resources, such as scaling factor optimization, alignment optimization and system combination. All methods are evaluated on two of the largest sign language corpora available.     

Realization of sign language motion using a dual-arm/hand humanoid robot

The recent increase in technological maturity has empowered robots to assist humans and provide daily services. Voice command usually appears as a popular human–machine interface for communication. Unfortunately, deaf people cannot exchange information from robots through vocal modalities. To interact with deaf people effectively and intuitively, it is desired that robots, especially humanoids, have manual communication skills, such as performing sign languages. Without ad hoc programming to generate a particular sign language motion, we present an imitation system to teach the humanoid robot performing sign languages by directly replicating observed demonstration. The system symbolically encodes the information of human hand–arm motion from low-cost depth sensors as a skeleton motion time-series that serves to generate initial robot movement by means of perception-to-action mapping. To tackle the body correspondence problem, the virtual impedance control approach is adopted to smoothly follow the initial movement, while preventing potential risks due to the difference in the physical properties between the human and the robot, such as joint limit and self-collision. In addition, the integration of the leg-joints stabilizer provides better balance of the whole robot. Finally, our developed humanoid robot, NINO, successfully learned by imitation from human demonstration to introduce itself using Taiwanese Sign Language.