A Multi-User 3-D Virtual Environment with Interactive Collaboration and Shared Whiteboard Technologies

A multi-user 3-D virtual environment allows remote participants to have a transparent communication as if they are communicating face-to-face. The sense of presence in such an environment can be established by representing each participant with a vivid human-like character called an avatar. We review several immersive technologies, including directional sound, eye gaze, hand gestures, lip synchronization and facial expressions, that facilitates multimodal interaction among participants in the virtual environment using speech processing and animation techniques. Interactive collaboration can be further encouraged with the ability to share and manipulate 3-D objects in the virtual environment. A shared whiteboard makes it easy for participants in the virtual environment to convey their ideas graphically. We survey various kinds of capture devices used for providing the input for the shared whiteboard. Efficient storage of the whiteboard session and precise archival at a later time bring up interesting research topics in information retrieval.     

Multimodal behavior realization for embodied conversational agents

Applications with intelligent conversational virtual humans, called Embodied Conversational Agents (ECAs), seek to bring human-like abilities into machines and establish natural human-computer interaction. In this paper we discuss realization of ECA multimodal behaviors which include speech and nonverbal behaviors. We devise RealActor, an open-source, multi-platform animation system for real-time multimodal behavior realization for ECAs. The system employs a novel solution for synchronizing gestures and speech using neural networks. It also employs an adaptive face animation model based on Facial Action Coding System (FACS) to synthesize face expressions. Our aim is to provide a generic animation system which can help researchers create believable and expressive ECAs.     

Presence and interaction in mixed reality environments

In this paper, we present a simple and robust mixed reality (MR) framework that allows for real-time interaction with virtual humans in mixed reality environments under consistent illumination. We will look at three crucial parts of this system: interaction, animation and global illumination of virtual humans for an integrated and enhanced presence. The interaction system comprises of a dialogue module, which is interfaced with a speech recognition and synthesis system. Next to speech output, the dialogue system generates face and body motions, which are in turn managed by the virtual human animation layer. Our fast animation engine can handle various types of motions, such as normal key-frame animations, or motions that are generated on-the-fly by adapting previously recorded clips. Real-time idle motions are an example of the latter category. All these different motions are generated and blended on-line, resulting in a flexible and realistic animation. Our robust rendering method operates in accordance with the previous animation layer, based on an extended for virtual humans precomputed radiance transfer (PRT) illumination model, resulting in a realistic rendition of such interactive virtual characters in mixed reality environments. Finally, we present a scenario that illustrates the interplay and application of our methods, glued under a unique framework for presence and interaction in MR.     

Semantic framework for interactive animation generation and its application in virtual shadow play performance

Designing and creating complex and interactive animation is still a challenge in the field of virtual reality, which has to handle various aspects of functional requirements (e.g. graphics, physics, AI, multimodal inputs and outputs, and massive data assets management). In this paper, a semantic framework is proposed to model the construction of interactive animation and promote animation assets reuse in a systematic and standardized way. As its ontological implementation, two domain-specific ontologies for the hand-gesture-based interaction and animation data repository have been developed in the context of Chinese traditional shadow play art. Finally, prototype of interactive Chinese shadow play performance system using deep motion sensor device is presented as the usage example.     

Design of an emotional and social interaction paradigm for the animation of 3D characters: the case of a therapy for brain injured people (the mirror neuron paradigm)

We propose an emotional and social interaction paradigm for the behavioural animation of virtual characters. Our research focuses on the production of verbal and paraverbal interactions between a human and a virtual character in the context of virtual therapy. The aim of this project is to assess virtual therapies for the rehabilitation of people with brain injuries caused by cerebral vascular accidents. The therapeutic paradigm is based on the neuroscience concept of “mirror neurones” which emphasise the perceptual side in the process of recovering neural damage. In our paradigm, the patient interacts with a virtual therapist (embodied autonomous agent) that guides and encourages him in order to fulfil the drills. Furthermore and following the “mirror neurone paradigm”, the patient can only see his virtual hands (which are handicapped). The virtual platform provides the patient with a vision of his hands undertaking a movement that he cannot perform. Doing so, we expect that the perception of this virtual movement may improve the recuperation (mirror neurone paradigm). In this paper, we also introduce the notion of an “intelligent emotional character” to produce a sense of social as well as emotional immersion in order to improve the interaction between the patient and the virtual therapist. We discuss the role of emotional interaction in the context of virtual social interaction and in particular its reflexive dimension. Finally, we present an architecture for the animation of virtual characters based on a multidisciplinary approach to model the emotional and social interaction.     

Personalization in an interactive learning environment through a virtual character

Traditional hypermedia applications present the same content and provide identical navigational support to all users. Adaptive Hypermedia Systems (AHS) make it possible to construct personalized presentations to each user, according to preferences and needs identified. We present in this paper an alternative approach to educational AHS where a virtual character personalizes the interaction with the user through the use of a particular recommender system. The character has natural language communication abilities; it can learn students’ profiles and use this knowledge to recommend appropriate contents and activities. Through its interaction with the user, the character is able to collect and organize information about students in order to identify appropriate suggestions of contents. The recommender system employs a knowledge representation scheme that is easy to understand and to modify, enabling teachers/tutors to explore the types of recommendations being made and to appreciate why they are made. An experiment with computer science students was carried out in order to validate the approach proposed. The results of the experiment showed that the presentation of personalized links through a virtual character had a positive impact in the users’ perception of the system as a learning tool. The combination of the virtual character with a recommender system proved to be a good alternative for the delivery of personalized contents without making constant changes in the main user interface. This approach provides mechanisms to guide users through paths of study followed by students with similar profiles, without violating the human–computer interaction principle of perceived stability.     

基于Forge云的艺用人体解剖绘画仿真系统

针对目前人物动画制作以及绘画过程中对人体结构认识不足的问题,以及虚拟现实技术在教育中的优势,提出了一个基于Forge云的艺用人体解剖绘画仿真系统方案.系统按照人体结构比例,采用块面加线的模式完成人物模型的构建及可视化过程,完全遵照动画运动规律,以骨骼动画结合三维动作捕捉的方式实现人体运动仿真.通过Forge云平台和Three.js完成人机交互.最后,将漫画模块和Forge云模块双向通信,完成漫画人物姿态仿真.通过测试证明,该系统的仿真度和易用性较高,为数字化学习和移动学习提供了环境,有助于学习者深入理解人体解剖结构,并正确掌握漫画人物的造型方法.     

虚拟人情绪行为动画模型

近年来 ,虚拟人行为动画已经成为计算机动画一个新的分枝 ,已往的研究多集中在虚拟人局部的情绪表达动画方面 ,如脸部动画 ,而对于给定的虚拟场景 ,则尚未考虑情绪产生的原因 .情绪是虚拟人和虚拟环境交互作用的结果 ,然而在计算机动画领域 ,虚拟人的情绪至今尚未得到清楚的描述 .为此 ,依据心理学的理论 ,提出了虚拟人情绪行为的动画模型 ,即 ,首先提出了情绪集合和情绪表达集合的概念 ,并建立了从情绪状态到情绪表达之间的映射 ;其次 ,着重分析了情绪产生的原因 ,并引入了情绪源的概念 ,如果一种情绪刺激的强度大于情绪的抵抗强度 ,那么这种情绪就会产生 ;此外 ,情绪状态可以采用有限状态机来描述 ,据此提出了情绪的变化流程 ;最后 ,在微机上通过调用 Microsoft Direct3D API,实现了虚拟人的情绪行为动画     

Animation Using Temporal Constraints: An Overview of the Animus System

Algorithm animation has a growing role in computer-aided algorithm design, documentation, and debugging, because interactive graphics is a richer channel than text for communication. Most animation is currently done laboriously by hand, and it often has the character of canned demonstrations with restricted user interaction. Animus is a system that allows for easy construction of an animation with minimal concern for lower-level graphics programming. Constraints are used to describe the appearance and structure of a picture as well as how those pictures evolve in time. The implementation and support of temporal constraints are substantive extensions to previous constraint languages that had only allowed for the specification of a static state. Use of the Animus system is demonstrated in the creation of animations of dynamic mechanical and electrical circuit simulations, sorting algorithms, problems in operating systems, and geometric curve drawing algorithms.     

Agent system using multimodal interfaces for a smart office environment

This paper presents an experimental study on an agent system with multimodal interfaces for a smart office environment. The agent system is based upon multimodal interfaces such as recognition modules for both speech and pen-mouse gesture, and identification modules for both face and fingerprint. For essential modules, speech recognition and synthesis were basically used for a virtual interaction between user and system. In this study, a real-time speech recognizer based on a Hidden Markov Network (HM-Net) was incorporated into the proposed system. In addition, identification techniques based on both face and fingerprint were adopted to provide a specific user with the service of a user-customized interaction with security in an office environment. In evaluation, results showed that the proposed system was easy to use and would prove useful in a smart office environment, even though the performance of the speech recognizer was not satisfactory mainly due to noisy environments.     

一种具有逼真效果的虚拟人动画生成方法

针对传统的虚拟人动画逼真度低、动画生成复杂、且动画存在飘移现象的问题,提出了一种基于视频关键帧获取运动数据的虚拟人动画生成方法。首先利用线性混合蒙皮算法建立了新的人体几何模型;其次利用正向运动学方法驱动虚拟人的骨骼模型,并利用平移补偿原则消除虚拟人运动中的飘移现象;最终实现了基于双目正交视觉获取关节运动参数的虚拟人动画。实验结果表明,新方法能逼真地模拟真实人体动作,且方法易于实现、实用性强。     

An inclusive Conformal Geometric Algebra GPU animation interpolation and deformation algorithm

In the last years, Geometric Algebra with its Euclidean, Homogeneous and Conformal models attracts the research interest in many areas of Computer Science and Engineering and particularly in Computer Graphics as it is shown that they can produce more efficient and smooth results than other algebras. In this paper, we present an all-inclusive algorithm for real-time animation interpolation and GPU-based geometric skinning of animated, deformable virtual characters using the Conformal model of Geometric Algebra (CGA). We compare our method with standard quaternions, linear algebra matrices and dual-quaternions blending and skinning algorithms and we illustrate how our CGA-GPU inclusive skinning algorithm can provide as smooth and more efficient results as state-of-the-art previous methods. Furthermore, the elements of CGA that handle transformations (CGA motors) can support translation, rotation and dilation(uniform scaling) of joints under a single, GPU-supported mathematical framework and avoid conversion between different mathematical representations in contrast to quaternions and dual-quaternions that support only rotation and rotation–translation, respectively. Hence, our main novelty is the replacement of different types of algebras, and their in-between conversions between CPU and GPU, such as linear algebra matrices, quaternions, dual-quaternions and Euler angles for animation interpolation and skinning with a single mathematical representation, the CGA motors which can optimally handle the composition of translation, rotation and scaling joint transformations and interpolations. Employing latest CGA code generators, we provide a sample implementation of our algorithm running natively in a vertex shader program on modern GPUs for typical deformable virtual character simulations.     

基于智能体的群体动画创作方法的研究与实现

当前,动画研究日益着重角色的智能和动作的细腻,群体动画系统的各模块之间耦合度日益提高,致使动画创作难度日益加大,尤其当智能角色数目增多时,其计算量将呈非线性快速增加,极大地影响了动画创作的实时性,无法在简单性和高效性方面较好地均衡。该文基于Steering行为理论,综合传统的智能角色动画和蒙皮骨骼动画技术,提出了一个基于自主智能体的群体动画角色模型。实例证明：此种模型制作简单,模块耦合度低且能通过很少的计算量实现实时交互,提高了动画创作效率,具有良好的应用前景。     

Self adaptive animation based on user perspective

In this paper we present a new character animation technique in which the animation adapts itself based on the change in the user’s perspective, so that when the user moves and their point of viewing the animation changes, then the character animation adapts itself in response to that change. The resulting animation, generated in real-time, is a blend of key animations provided a priori by the animator. The blending is done with the help of efficient dual-quaternion transformation blending. The user’s point of view is tracked using either computer vision techniques or a simple user-controlled input modality, such as mouse-based input. This tracked point of view is then used to suitably select the blend of animations. We show a way to author and use such animations in both virtual as well as augmented reality scenarios and demonstrate that it significantly heightens the sense of presence for the users when they interact with such self adaptive animations of virtual characters.     

A simple footskate removal method for virtual reality applications

Footskate is a common problem encountered in interactive applications dealing with virtual character animations. It has proven difficult to fix without the use of complex numerical methods, which require expert skills for their implementations, along with a fair amount of user interaction to correct a motion. On the other hand, deformable bodies are being increasingly used in virtual reality (VR) applications, allowing users to customize their avatar as they wish. This introduces the need of adapting motions without any help from a designer, as a random user seldom has the skills required to drive the existing algorithms towards the right solution. In this paper, we present a simple method to remove footskate artifacts in VR applications. Unlike previous algorithms, our approach does not rely on the skeletal animation to perform the correction but rather on the skin. This ensures that the final foot planting really matches the virtual character’s motion. The changes are applied to the root joint of the skeleton only so that the resulting animation is as close as possible to the original one. Eventually, thanks to the simplicity of its formulation, it can be quickly and easily added to existing frameworks.     

支持多模即时控制的角色动画库

该文主要介绍了一个支持多模即时控制的角色动画库,包括其整体架构、脚本格式和动画引擎。提出了一种通过以XML作为主要格式的角色定义,将角色、骨架和动作定义相分离以提高动画库的复用性的方法。采用动作有限状态机使得系统能够通过一系列的动画指令来控制动画角色的行为,简化了对动画角色的控制。同时系统通过支持多模输入并在虚拟化身上做出即时反应,增强了虚拟现实应用中的沉浸感、临场参与感和交互性。     

严肃游戏中虚拟角色行为建模综述

严肃游戏是计算机游戏一个新的发展方向,可以提供形象互动的模拟教学环境,已经广泛应用于科学教育、康复医疗、应急管理、军事训练等领域。虚拟角色是严肃游戏中模拟具有生命特征的图形实体,行为可信的虚拟角色能够提升用户使用严肃游戏的体验感。严肃游戏中的图形渲染技术已经逐步成熟,而虚拟角色行为建模的研究尚在初级阶段。可信的虚拟角色必须能够具有感知、情绪和行为能力。本文分别从游戏剧情与行为、行为建模方法、行为学习和行为建模评价等4个方面来分析虚拟角色行为建模研究。分析了有限状态机和行为树的特点,讨论了虚拟角色的行为学习方法。指出了强化学习的关键要素,探讨了深度强化学习的应用途径。综合已有研究,归纳了虚拟角色行为框架,该框架主要包括感觉输入、知觉分析、行为决策和动作4大模块。从情感计算的融入、游戏剧情和场景设计、智能手机平台和多通道交互4个角度讨论需要进一步研究的问题。虚拟角色的行为建模需要综合地考虑游戏剧情、机器学习和人机交互技术,构建具有自主感知、情绪、行为、学习能力、多通道交互的虚拟角色能够极大地提升严肃游戏的感染力,更好地体现寓教于乐。     

Physically Based Sound Synthesis for Large-Scale Virtual Environments

By offering a natural, intuitive interface with the virtual world, auditory display can enhance a user's experience in a multimodal virtual environment and further improve the user's sense of presence. However, compared to graphical display, sound synthesis has not been well investigated because of the extremely high computational cost for simulating realistic sounds. The state of the art for sound production in virtual environments is to use recorded sound clips that events in the virtual environment trigger, similar to how recorded animation sequences in the earlier days generated all the character motion in the virtual world. In this article, we describe several techniques for accelerating sound simulation, thereby enabling realistic, physically based sound synthesis for large-scale virtual environments     

Silk road dance exhibition system based on motion capture

Silk Road dance is an excellent traditional Chinese dance drama, which has high artistic values. This article introduces the virtual performance techniques of Silk Road based on motion capture. First, obtain the dance data using motion capture technique according to the features of dance itself, then establish figure model, finally combines these data to make 3-D animation system to perform Silk Road, meanwhile designs some interfaces to introduce it. This system excavates details of the dance, and can be reference of man-machine interaction and animation production.     

基于动捕数据驱动的三维民族舞蹈展示方法

三维模型动画在数字化设计和应用中具有重要意义,受到越来越多研究者关注;但如何通过三维数字化原真再现民族舞蹈表演是极具挑战的问题.本论文通过动捕技术采集舞蹈动作实现舞蹈数字化展示.具体方法是:首先利用动捕设备捕获人体动作数据,然后在Maya中进行人物建模、骨骼绑定、蒙皮和权重调节,再通过MotionBuilder将3D模型与动捕数据结合,最终完成了现实舞蹈动作的虚拟人展演.论文构建了一个面向民族舞蹈展演的虚拟场景,并以13个民族的舞蹈为数字化内容,推广动捕驱动的舞蹈展演方法的应用.