Educational resources and implementation of a Greek sign language synthesis architecture

In this paper, we present how creation and dynamic synthesis of linguistic resources of Greek Sign Language (GSL) may serve to support development and provide content to an educational multitask platform for the teaching of GSL in early elementary school classes. The presented system utilizes standard virtual character (VC) animation technologies for the synthesis of sign sequences/streams, exploiting digital linguistic resources of both lexicon and grammar of GSL. Input to the system is written Greek text, which is transformed into GSL and animated on screen. To achieve this, a syntactic parser decodes the structural patterns of written Greek and matches them into equivalent patterns of GSL, which are then signed by a VC. The adopted notation system for the representation of GSL phonology incorporated in the system’s lexical knowledge database, is Hamburg Notation System (HamNoSys). For the implementation of the virtual signer tool, the definition of the VC follows the h-anim standard and is implemented in a web browser using a standard VRML plug-in.     

A Chinese sign language recognition system based on SOFM/SRN/HMM

In sign language recognition (SLR), the major challenges now are developing methods that solve signer-independent continuous sign problems. In this paper, SOFM/HMM is first presented for modeling signer-independent isolated signs. The proposed method uses the self-organizing feature maps (SOFM) as different signers' feature extractor for continuous hidden Markov models (HMM) so as to transform input signs into significant and low-dimensional representations that can be well modeled by the emission probabilities of HMM. Based on these isolated sign models, a SOFM/SRN/HMM model is then proposed for signer-independent continuous SLR. This model applies the improved simple recurrent network (SRN) to segment continuous sign language in terms of transformed SOFM representations, and the outputs of SRN are taken as the HMM states in which the lattice Viterbi algorithm is employed to search the best matched word sequence. Experimental results demonstrate that the proposed system has better performance compared with conventional HMM system and obtains a word recognition rate of 82.9% over a 5113-sign vocabulary and an accuracy of 86.3% for signer-independent continuous SLR.     

A multi-class classification strategy for Fisher scores: Application to signer independent sign language recognition

Fisher kernels combine the powers of discriminative and generative classifiers by mapping the variable-length sequences to a new fixed length feature space, called the Fisher score space. The mapping is based on a single generative model and the classifier is intrinsically binary. We propose a multi-class classification strategy that applies a multi-class classification on each Fisher score space and combines the decisions of multi-class classifiers. We experimentally show that the Fisher scores of one class provide discriminative information for the other classes as well. We compare several multi-class classification strategies for Fisher scores generated from the hidden Markov models of sign sequences. The proposed multi-class classification strategy increases the classification accuracy in comparison with the state of the art strategies based on combining binary classifiers. To reduce the computational complexity of the Fisher score extraction and the training phases, we also propose a score space selection method and show that, similar or even higher accuracies can be obtained by using only a subset of the score spaces. Based on the proposed score space selection method, a signer adaptation technique is also presented that does not require any re-training.     

A linguistic ontology of space for natural language processing

We present a detailed semantics for linguistic spatial expressions supportive of computational processing that draws substantially on the principles and tools of ontological engineering and formal ontology. We cover language concerned with space, actions in space and spatial relationships and develop an ontological organization that relates such expressions to general classes of fixed semantic import. The result is given as an extension of a linguistic ontology, the Generalized Upper Model, an organization which has been used for over a decade in natural language processing applications. We describe the general nature and features of this ontology and show how we have extended it for working particularly with space. Treaitng the semantics of natural language expressions concerning space in this way offers a substantial simplification of the general problem of relating natural spatial language to its contextualized interpretation. Example specifications based on natural language examples are presented, as well as an evaluation of the ontology's coverage, consistency, predictive power, and applicability.     

基于手势数据分析的地方手语动画合成方法研究*

为了在现有的手势数据基础上合成新的地方手语手势,采取对这些手势数据进行运动相似性分段和动态聚类自动获取具有时序特征的基本手形类数据以及结合手臂状态数据的方法,最后生成新的地方手语手势动画数据。这不仅节省了重新采集数据需要的昂贵设备和大量工作,而且也能达到手势表达准确性的目的。     

A prototype Greek text to Greek Sign Language conversion system

In this article, a prototype Greek text to Greek Sign Language (GSL) conversion system is presented. The system is integrated into an educational platform that addresses the needs of teaching GSL grammar and was developed within the SYNENNOESE project (Efthimiou et al. 2004a Efthimiou, E. 2004a. “Developing an e-learning platform for the Greek sign language”. In Computer helping people with special needs, Lecture Notes in Computer Science Edited by: Miesenberger, K., Klaus, J. and Zagler, W. Vol. 3118, 1107–1113.  [Google Scholar]. Developing an e-learning platform for the Greek sign language. In: K. Miesenberger, J. Klaus, and W. Zagler, eds. Computer helping people with special needs, Lecture Notes in Computer Science, Springer-Verlag, vol. 3118, 1107–1113). The detailed implementation of the language-processing component of a Greek text to GSL conversion system is provided, focusing upon the inherent problems of knowledge elicitation of sign language (SL) grammar and its implementation within a parser framework. It is based on an SL dictionary (Efthimiou et al. 2004a Efthimiou, E. 2004a. “Developing an e-learning platform for the Greek sign language”. In Computer helping people with special needs, Lecture Notes in Computer Science Edited by: Miesenberger, K., Klaus, J. and Zagler, W. Vol. 3118, 1107–1113.  [Google Scholar]) database of coded GSL knowledge. The proposed system has been designed and implemented after considering most state-of-the-art SL machine translation or Conversion systems, such as Vsigns (Papadogiorgaki et al. 2004 Papadogiorgaki, M. VSigns – a virtual sign synthesis web tool. Proceedings of Sixth COST 276 Workshop on Information and Knowledge Management for Integrated Media Communication. May2004, Thessaloniki, Greece.  [Google Scholar]. VSigns – a virtual sign synthesis web tool. In: Proceedings of Sixth COST 276 Workshop on Information and Knowledge Management for Integrated Media Communication, May 2004, Thessaloniki, Greece), ZARDOZ (Veale et al. 1998 Veale, T., Conway, A. and Collins, B. 1998. The challenges of cross-modal translation: English to sign language translation in the ZARDOZ system. Machine Translation, 13: 81–106. [Crossref] , [Google Scholar]. The challenges of cross-modal translation: English to sign language translation in the ZARDOZ system. Machine Translation, 13, 81–106) and SignSynth (Angus 2001 Angus, G. S.B. 2001. “SignSynth: a sign language synthesis application using Web3D and Perl”. In Gesture and Sign Language in Human–Computer Interaction, London, , UK: International Gesture Workshop.  [Google Scholar]. SignSynth: a sign language synthesis application using Web3D and Perl. In: Gesture and Sign Lanauage in Human–Computer Interaction. London, UK: International Gesture Workshop), and taking into account their advantages and disadvantages. The overall architecture is innovative since other existing systems either do not consider the GSL or they cannot be effectively applied on sentences but just on single words. The system is demonstrable on any conventional PC.     

A knowledge-based Digital Shadow for machining industry in a Digital Twin perspective

This paper addresses the problems of data management and analytics for decision-aid by proposing a new vision of Digital Shadow (DS) which would be considered as the core component of a future Digital Twin. Knowledge generated by experts and artificial intelligence, is transformed into formal business rules and integrated into the DS to enable the characterization of the real behavior of the physical system throughout its operation stage. This behavior model is continuously enriched by direct or derived learning, in order to improve the digital twin. The proposed DS relies on data analytics (based on unsupervised learning) and on a knowledge inference engine. It enables the incidents to be detected and it is also able to decipher its operational context. An example of this application in the aeronautic machining industry is provided to stress both the feasibility of the proposition and its potential impact on shop floor performance.     

Dynamics-based analysis and synthesis of human locomotion

One of the best ways to synthesize realistic human motions is to animate characters from captured motion data that inherently respect motion laws. Retargeting and interpolation methods are often used to adapt these motions to different representations of the character and to various environmental constraints but they may introduce physical inaccuracies, although the synthesized motions are natural looking. This paper presents a method for evaluating the physical correctness of retargeted and interpolated locomotions using an inverse dynamics analysis. Furthermore, we propose to improve an initial database with analysed motions that are synthesized again by using a forward dynamics approach. The analysis algorithm consists in determining the resulting forces and torques at joints. With this intention, we develop an automatic creation process of the mass/inertia model of the character. Then using support phase recognition, we compute resulting forces and torques by an inverse dynamics method. The retargeting and the interpolation methods change the physics of the motions. This change is evaluated by using the results of our analysis on artificial and real motions and by using literature results and experimental data from force plates. The evaluation relies on the study of several retargeting and interpolation parameters such as the global size of the character or the structure of the model. The output of this evaluation, the resulting forces and torques at joints, are used to produce physically valid motions by using forward dynamics simulation. With this purpose, we introduce forces and torques normalizations, and finally the synthesized motions may improve the initial database.     

Towards subject independent continuous sign language recognition: A segment and merge approach

This paper presents a segment-based probabilistic approach to robustly recognize continuous sign language sentences. The recognition strategy is based on a two-layer conditional random field (CRF) model, where the lower layer processes the component channels and provides outputs to the upper layer for sign recognition. The continuously signed sentences are first segmented, and the sub-segments are labeled SIGN or ME (movement epenthesis) by a Bayesian network (BN) which fuses the outputs of independent CRF and support vector machine (SVM) classifiers. The sub-segments labeled as ME are discarded and the remaining SIGN sub-segments are merged and recognized by the two-layer CRF classifier; for this we have proposed a new algorithm based on the semi-Markov CRF decoding scheme. With eight signers, we obtained a recall rate of 95.7% and a precision of 96.6% for unseen samples from seen signers, and a recall rate of 86.6% and a precision of 89.9% for unseen signers.     

Retrieval of online handwriting by synthesis and matching

Search and retrieval is gaining importance in the ink domain due to the increase in the availability of online handwritten data. However, the problem is challenging due to variations in handwriting between various writers, digitizers and writing conditions. In this paper, we propose a retrieval mechanism for online handwriting, which can handle different writing styles, specifically for Indian languages. The proposed approach provides a keyboard-based search interface that enables to search handwritten data from any platform, in addition to pen-based and example-based queries. One of the major advantages of this framework is that information retrieval techniques such as ranking relevance, detecting stopwords and controlling word forms can be extended to work with search and retrieval in the ink domain. The framework also allows cross-lingual document retrieval across Indian languages.     

A belief-based sequential fusion approach for fusing manual signs and non-manual signals

Most of the research on sign language recognition concentrates on recognizing only manual signs (hand gestures and shapes), discarding a very important component: the non-manual signals (facial expressions and head/shoulder motion). We address the recognition of signs with both manual and non-manual components using a sequential belief-based fusion technique. The manual components, which carry information of primary importance, are utilized in the first stage. The second stage, which makes use of non-manual components, is only employed if there is hesitation in the decision of the first stage. We employ belief formalism both to model the hesitation and to determine the sign clusters within which the discrimination takes place in the second stage. We have implemented this technique in a sign tutor application. Our results on the eNTERFACE’06 ASL database show an improvement over the baseline system which uses parallel or feature fusion of manual and non-manual features: we achieve an accuracy of 81.6%.     

A learning-based synthesis approach todecentralized supervisory control of discrete event systems with unknown plants

In this paper, we consider the problem of automatic synthesis of decentralized supervisor for uncertain discrete event systems. In particular, we study the case when the uncontrolled plant is unknown a priori. To deal with the unknown plants, we first characterize the conormality of prefix-closed regular languages and propose formulas for computing the supremal conormal sublanguages; then sufficient conditions for the existence of decentralized supervisors are given in terms of language controllability and conormality and a learning-based algorithm to synthesize the supervisor automatically is proposed. Moreover, the paper also studies the on-line decentralized supervisory control of concurrent discrete event systems that are composed of multiple interacting unknown modules. We use the concept of modular controllability to characterize the necessary and sufficient conditions for the existence of the local supervisors, which consist of a set of local supervisor modules, one for each plant module and which determines its control actions based on the locally observed behaviors, and an on-line learning-based local synthesis algorithm is also presented. The correctness and convergence of the proposed algorithms are proved, and their implementation are illustrated through examples.     

Language use: A performance model

A model for language use or performance is given which is essentially independent of the standard theories for language acquisition and competence (the knowledge of language). Comparisons with computer models for language understanding are provided. It is suggested that basic research in human language comprehension must precede construction of new artificial intelligence models.     

A design methodology for form-based knowledge reuse and representation

Paper forms are regularly used for collecting and disseminating knowledge in offices; they are a natural way of eliciting requirements of knowledge workers. Many organizations have implemented a groupware system to integrate the organizational knowledge and support knowledge creation. However, design methods for flexible form-based knowledge reuse and representation are limited. We developed a methodology based on the enhanced cognitive fit theory; it utilizes factoring and synthesis principles to manipulate form-based knowledge. The methodology was articulated using the design science research methodology. A prototype embedded methodology was built to support a knowledge worker in knowledge creation and reuse in a high tech firm. The resulting system allowed flexible form-based knowledge creation that was useful for problem solving and exploiting opportunities. Implications and conclusions are discussed.     

资源环境综合模型的集成分析

针对资源环境问题的复杂性和综合性,在对海量资源环境模型进行分析并构建了资源环境模型库系统(REMS)的基础上,运用综合集成的方法论理念,根据复杂性资源环境问题综合模拟分析过程中系列子模型各自所承担的模拟功能和重要性,从资源环境学科背景的研究角度,探索性地提出了资源环境综合模型的串联集成、并联集成、混联集成等三种通用的综合集成模式。以HLZ生态系统时空变化趋势及情景分析模型集成为例,对集成模式的有效性和可行性进行了论证分析。