Educational resources and implementation of a Greek sign language synthesis architecture

In this paper, we present how creation and dynamic synthesis of linguistic resources of Greek Sign Language (GSL) may serve to support development and provide content to an educational multitask platform for the teaching of GSL in early elementary school classes. The presented system utilizes standard virtual character (VC) animation technologies for the synthesis of sign sequences/streams, exploiting digital linguistic resources of both lexicon and grammar of GSL. Input to the system is written Greek text, which is transformed into GSL and animated on screen. To achieve this, a syntactic parser decodes the structural patterns of written Greek and matches them into equivalent patterns of GSL, which are then signed by a VC. The adopted notation system for the representation of GSL phonology incorporated in the system’s lexical knowledge database, is Hamburg Notation System (HamNoSys). For the implementation of the virtual signer tool, the definition of the VC follows the h-anim standard and is implemented in a web browser using a standard VRML plug-in.     

非特定人手语数据生成及其有效性检测

根据手势手语的特点,提出了手语语言学和人体运动学相结合的非特定人手语数据的生成和检测方法.首先,Mean-Shift算法有控制生成强度的优点,将改进的Mean-Shift算法应用于手形数据通道的生成,以保持手势手语的语言学特性,并应用关键手形的音韵标记进行数据有效性的检测;其次,为了丰富手语手势动作的运动特性,将改进的遗传算法应用于与运动相关的数据通道进行数据生成,并应用拉班舞谱对其进行数据有效性检测;最后,提出了基于原始样本的检测实验框架,使得所提出的检测方法适用于语言类的多类别数据检测问题.实验结果表明,所提出的非特定人手语数据的生成和检测方法是有效的.     

Dynamic–static unsupervised sequentiality,statistical subunits and lexicon for sign language recognition

We introduce a new computational phonetic modeling framework for sign language (SL) recognition. This is based on dynamic–static statistical subunits and provides sequentiality in an unsupervised manner, without prior linguistic information. Subunit “sequentiality” refers to the decomposition of signs into two types of parts, varying and non-varying, that are sequentially stacked across time. Our approach is inspired by the Movement–Hold SL linguistic model that refers to such sequences. First, we segment signs into intra-sign primitives, and classify each segment as dynamic or static, i.e., movements and non-movements. These segments are then clustered appropriately to construct a set of dynamic and static subunits. The dynamic/static discrimination allows us employing different visual features for clustering the dynamic or static segments. Sequences of the generated subunits are used as sign pronunciations in a data-driven lexicon. Based on this lexicon and the corresponding segmentation, each subunit is statistically represented and trained on multimodal sign data as a hidden Markov model. In the proposed approach, dynamic/static sequentiality is incorporated in an unsupervised manner. Further, handshape information is integrated in a parallel hidden Markov modeling scheme. The novel sign language modeling scheme is evaluated in recognition experiments on data from three corpora and two sign languages: Boston University American SL which is employed pre-segmented at the sign-level, Greek SL Lemmas, and American SL Large Vocabulary Dictionary, including both signer dependent and unseen signers' testing. Results show consistent improvements when compared with other approaches, demonstrating the importance of dynamic/static structure in sub-sign phonetic modeling.     

非特定人手语数据生成及其有效性检测

根据手势手语的特点,提出了手语语言学和人体运动学相结合的非特定人手语数据的生成和检测方法. 首先,Mean-Shift 算法有控制生成强度的优点,将改进的Mean-Shift 算法应用于手形数据通道的生成,以保持手势手 语的语言学特性,并应用关键手形的音韵标记进行数据有效性的检测;其次,为了丰富手语手势动作的运动特性,将 改进的遗传算法应用于与运动相关的数据通道进行数据生成,并应用拉班舞谱对其进行数据有效性检测;最后,提出 了基于原始样本的检测实验框架,使得所提出的检测方法适用于语言类的多类别数据检测问题.实验结果表明,所提 出的非特定人手语数据的生成和检测方法是有效的.     

基于VRML虚拟人的维文手语库的构建

维吾尔语的手语合成有助于改善维吾尔族聋哑人与听力正常人进行自然交流,也可以应用于计算机辅助维吾尔哑语教学、维文电视节目播放等方面。维文手语库是维吾尔语手语合成的基础。通过分析维吾尔手语的特点,采用关键帧插值技术来控制VRML虚拟人的手势动作,利用Visual C++和OpenGL环境实现了一个维吾尔文的手势编辑系统,通过手势运动数据驱动虚拟人来实时显示当前的手势状态。通过该系统,收集了常用的维吾尔语词汇及32个维吾尔字母的手势运动数据。     

基于Ontology的英汉机器翻译研究

高质量的机器翻译(Machine Translation)系统必须充分结合语言学知识以及语言中性的世界知识。近年来,ontology被广泛用于在概念层对世界知识建模,本文介绍一个基于ontology的英汉机器翻译模型系统,在这个系统中,ontology作为世界知识的模型,它是通过把概念组织成一个层次结构并同时在概念间建立丰富的概念联系而构成的。通过把某种语言中的词汇映射到ontology中的概念,可以支持在源语言分析时进行歧义消解和目标语生成时的词汇选择,并可以作为源语言和目的语言之间的中介表示的概念来源。在系统中,中介表示是用概念图(Conceptual Graph)来表示的。     

An integrated, dual learner for grammars and ontologies

We introduce a dual-use methodology for automating the maintenance and growth of two types of knowledge sources, which are crucial for natural language text understanding—background knowledge of the underlying domain and linguistic knowledge about the lexicon and the grammar of the underlying natural language. A particularity of this approach is that learning occurs simultaneously with the on-going text understanding process. The knowledge assimilation process is centered around the linguistic and conceptual ‘quality' of various forms of evidence underlying the generation, assessment and on-going refinement of lexical and concept hypotheses. On the basis of the strength of evidence, hypotheses are ranked according to qualitative plausibility criteria, and the most reasonable ones are selected for assimilation into the already given lexical class hierarchy and domain ontology.     

Collecting and evaluating the CUNY ASL corpus for research on American Sign Language animation

While there is great potential for sign language animation generation software to improve the accessibility of information for deaf individuals with low written-language literacy, the understandability of current sign language animation systems is limited. Data-driven methodologies using annotated sign language corpora encoding detailed human movement have enabled some researchers to address several key linguistic challenges in ASL generation. This article motivates and describes our current research on collecting a motion-capture corpus of American Sign Language (ASL). As an evaluation of our motion-capture configuration, calibration, and recording protocol, we have conducted several rounds of evaluation studies with native ASL signers, and we have made use of our collected data to synthesize novel animations of ASL, which have also been evaluated in experimental studies with native signers.     

An Assessment of Written/Interactive Dialogue for Information Retrieval Applications

For the foreseeable future, natural language access to databases and other information systems will require the exchange of written messages between the system and user. Such written/interactive dialogue is unique, having the qualities of both written text and spoken discourse yet, also, differing significantly from both. In the present study, we used "Wizard of Oz" techniques to elicit written/interactive dialogue for information retrieval purposes. Our objectives in doing this were (a) to assess the general nature and prevalence of linguistic and dialogue phenomena within the written/interactive register and (b) to determine the impact of user interface shortcuts, such as precanned messages and patterned output, on both the complexity of written/interactive dialogue and general measures of user satisfaction. Our findings indicate that written/interactive dialogue for information retrieval would be very feasible. In spite of slow system response times, subjective reactions from the users were positive, the size of the lexicon used in the dialogues was small, the dialogues decomposed readily into hierarchical structures, and the number and distribution of anaphors were also rather reasonable. Two independent variables were also manipulated: (a) the degree of prefamiliarization given to participants about the base of travel information they would be accessing (i.e., the primer variable) and (b) the degree of constraint on the Wizard's ability to formulate natural language responses to the user (i.e., the natural language output variable). Failure to provide either a primer or a realistic, human natural language output made the dialogues more complex in a number of different ways.     

The Challenges of Cross-Modal Translation: English-to-Sign-Language Translation in the Zardoz System

The sign languages used by deaf communities around the world represent a linguistic challenge that natural-language researchers in AI have only recently begun to take up. This challenge is particularly relevant to research in Machine Translation (MT), as natural sign languages have evolved in deaf communities into efficient modes of gestural communication, which differ from English not only in modality but in grammatical structure, exploiting a higher dimensionality of spatial expression. In this paper we describe Zardoz, an on-going AI research system that tackles the cross-modal MT problem, translating English text into fluid sign language. The paper presents an architectural overview of Zardoz, describing its central blackboard organization, the nature of its interlingual representation, and the major components which interact through this blackboard both to analyze the verbal input and generate the corresponding gestural output in one of a number of sign variants.     

Unsupervised Learning of a Chinese Spontaneous and Colloquial Speech Lexicon with Content and Filler Phrase Classification

There is significant lexical difference—words and usage of words-between spontaneous/colloquial language and the written language. This difference affects the performance of spoken language recognition systems that use statistical language models or context-free-grammars because these models are based on the written language rather than the spoken form. There are many filler phrases and colloquial phrases that appear solely or more often in spontaneous and colloquial speech. Chinese languages perhaps exemplify such a difference as many colloquial forms of the language, such as Cantonese, exist strictly in spoken forms and are different from the written standard Chinese, which is based on Mandarin. A conventional way of dealing with this issue is to add colloquial terms manually to the lexicon. However, this is time-consuming and expensive. Meanwhile, supervised learning requires manual tagging of large corpuses, which is also time-consuming. We propose an unsupervised learning method to find colloquial terms and classify filler and content phrases in spontaneous and colloquial Chinese, including Cantonese. We propose using frequency strength, and spread measures of character pairs and groups to extract automatically frequent, out-of-vocabulary colloquial terms to add to a standard Chinese lexicon. An unsegmented, and unannotated corpus is segmented with the augmented lexicon. We then propose a Markov classifier to classify Chinese characters into either content or filler phrases in an iterative training method. This method is task-independent and can extract even mixed language terms. We show the effectiveness of our method by both a natural language query processing task and an adaptive Cantonese language-modeling task. The precision for content phrase extraction and classification is around 80%, with a recall of 99%, and the precision for filler phrase extraction and classification is around 99.5% with a recall of approximately 89%. The web search precision using these extracted content words is comparable to that of the search results with content phrases selected by humans. We adapt a language model trained from written texts with the Hong Kong Newsgroup corpus. It outperforms both the standard Chinese language model and also the Cantonese language model. It also performs better than the language model trained a simply by concatenating two sets of standard and colloquial texts.     

SignMT: An alternative language learning tool

Learning a second language is very difficult, especially, for the disabled; the disability may be a barrier to learn and to utilize information written in text form. We present the SignMT, Thai sign to Thai machine translation system, which is able to translate from Thai sign language into Thai text. In the translation process, SignMT takes into account the differences between Thai and Thai sign language in terms of both syntax and semantic to ensure the accuracy of translation. SignMT was designed to be not only an automatic interpreter but also a language learning tool. It provides meaning of each word in both text and image forms which is easy to understand by the deaf. The grammar information and the order of the sentence are presented in order to help the deaf in learning Thai, their second language. With SignMT, deaf students are less dependent on a teacher, have more freedom to experiment with their own language, and improve their knowledge and learning skill.     

怎样用物性结构知识解决“网球问题”？

“网球问题”指怎样把racquet(网球拍)、ball(网球)和net(球网)之类具有情境联想关系的词汇概念联系起来、发现它们之间的语义和推理关系。这是一个自然语言处理和相关的语言知识资源建设的世界性难题。该文以求解“网球问题”为目标,对目前比较主流的几种语言词汇和概念知识库系统(包括WordNet、VerbNet、FrameNet、ConceptNet等)进行检讨,指出它们在解决“网球问题”上还都存在一定的局限性,着重分析它们为什么不能解决“网球问题”。进而指出基于生成词库论的名词物性结构知识描写体系可以解决“网球问题”,主张用名词的物性结构知识和相关的句法组合知识来构建一种以名词(实体)为核心的词汇概念网络,以弥补上述几种知识库系统的不足,为自然语言处理提供一种可资参考的词汇概念知识库体系。     

Voice-controlled modular fuzzy neural controller with enhanced user autonomy

In this article, a fuzzy neural network (FNN)-based approach is presented to interpret imprecise natural language (NL) commands for controlling a machine. This system, (1) interprets fuzzy linguistic information in NL commands for machines, (2) introduces a methodology to implement the contextual meaning of NL commands, and (3) recognizes machine-sensitive words from the running utterances which consist of both in-vocabulary and out-of-vocabulary words. The system achieves these capabilities through a FNN, which is used to interpret fuzzy linguistic information, a hidden Markov model-based key-word spotting system, which is used to identify machine-sensitive words among unrestricted user utterances, and a possible framework to insert the contextual meaning of words into the knowledge base employed in the fuzzy reasoning process. The system is a complete system integration which converts imprecise NL command inputs into their corresponding output signals in order to control a machine. The performance of the system specifications is examined by navigating a mobile robot in real time by unconditional speech utterances. This work was presented, in part, at the Seventh International Symposium on Artificial Life and Robotics, Oita, Japan, January 16–18, 2002     

Algebraic semantics for natural language: some philosophy, some applications

Information processing, when performed by an intelligent agent, draws on a wide array of knowledge sources. Among them are world knowledge, situation knowledge, conceptual knowledge and linguistic knowledge. The focus in this paper will be on the semantic knowledge which is part of the general linguistic competence of any speaker of a natural language (NL).In particular, this knowledge contains ways of organizing the linguistic ontology, i.e. the collection of heterogeneous entities that make up the domain of discourse. The representation language that is proposed here to model this knowledge stresses the structural properties of the ontology. This approach has been persued under the name of algebraic semantics.The paper starts out by explaining the term "algebraic semantics" as it is used in logic. Two senses of "algebraic" are distinguished that are called here "conceptual" and "structural". These two senses of the algebraic method are then applied to NL semantics. The conceptual part is realized by the method of structuring the domains of linguistic ontology in various ways. Thus plural entities are recognized along with mass entities and events. The common outlook here is mereological or lattice-theoretical. Some applications to the study of plurals are given that are to show the usefulness of the algebraic approach. Finally, the ontology of plurals is addressed, and comments are made on some relevant discussion of mereology in recent philosophical work. In sum, it is contended that the algebraic perspective while being of interest in semantics and philosophy proper, also fits both the spirit and the practice of much work that has been done in the Artificial Intelligence (AI) field of knowledge representation.     

维吾尔语KP_V句型的文法手语编辑系统的设计

自然语言书面表达通过文字符号载体来实现,聋哑学生经过专门的训练才能掌握理解和运用书面语言的能力。为实现基于虚拟人的手语合成技术的维吾尔文书面语用手语符号表达,对维吾尔语KP_V句型分析,提出了该句型的文本内容用手势语和手指语表示的转换方法,给出了维吾尔文法手语编辑系统的流程。系统通过虚拟人建模、手语库构建、姿态编辑,运用插值算法合成显示播放,实现了维吾尔文法手语编辑。该研究对基于拼音文字的维吾尔文本转换为混合手势语和手指的手语合成系统设计与实现具有参考意义。     

基于自裁剪异构图的NL2SQL模型

自然语言转换为结构化查询语言（NL2SQL）是语义解析领域的重要任务,其核心为对数据库模式和自然语言问句进行联合学习。现有研究通过将整个数据库模式和自然语言问句联合编码构建异构图,使得异构图中引入大量无用信息,并且忽略了数据库模式中不同信息的重要性。为提高NL2SQL模型的逻辑与执行准确率,提出一种基于自裁剪异构图与相对位置注意力机制的NL2SQL模型（SPRELA）。采用序列到序列的框架,使用ELECTRA预训练语言模型作为骨干网络。引入专家知识,对数据库模式和自然语言问句构建初步异构图。基于自然语言问句对初步异构图进行自裁剪,并使用多头相对位置注意力机制编码自裁剪后的数据库模式与自然语言问句。利用树型解码器和预定义的SQL语法,解码生成SQL语句。在Spider数据集上的实验结果表明,SPRELA模型执行准确率达到71.1%,相比于相同参数量级别的RaSaP模型提升了1.1个百分点,能够更好地将数据库模式与自然语言问句对齐,从而理解自然语言查询中的语义信息。