Real Time Animation of Virtual Humans: A Trade‐off Between Naturalness and Control

Virtual humans are employed in many interactive applications using 3D virtual environments, including (serious) games. The motion of such virtual humans should look realistic (or ‘natural’) and allow interaction with the surroundings and other (virtual) humans. Current animation techniques differ in the trade‐off they offer between motion naturalness and the control that can be exerted over the motion. We show mechanisms to parametrize, combine (on different body parts) and concatenate motions generated by different animation techniques. We discuss several aspects of motion naturalness and show how it can be evaluated. We conclude by showing the promise of combinations of different animation paradigms to enhance both naturalness and control.     

Path planning directed motion control of virtual humans in complex environments

Natural motion synthesis of virtual humans have been studied extensively, however, motion control of virtual characters actively responding to complex dynamic environments is still a challenging task in computer animation. It is a labor and cost intensive animator-driven work to create realistic human motions of character animations in a dynamically varying environment in movies, television and video games. To solve this problem, in this paper we propose a novel approach of motion synthesis that applies the optimal path planning to direct motion synthesis for generating realistic character motions in response to complex dynamic environment. In our framework, SIPP (Safe Interval Path Planning) search is implemented to plan a globally optimal path in complex dynamic environments. Three types of control anchors to motion synthesis are for the first time defined and extracted on the obtained planning path, including turning anchors, height anchors and time anchors. Directed by these control anchors, highly interactive motions of virtual character are synthesized by motion field which produces a wide variety of natural motions and has high control agility to handle complex dynamic environments. Experimental results have proven that our framework is capable of synthesizing motions of virtual humans naturally adapted to the complex dynamic environments which guarantee both the optimal path and the realistic motion simultaneously.     

Realistic Collision Avoidance of Upper Limbs Based on Neuroscience Models

When articulated figures interact in a 3D environment, collisions are highly likely and must often be avoided. We present a method automatically producing realistic collision-free animation of the upper arms. Based on the latest models of collision avoidance provided by neuroscience, our method allows realistic interpolation of keyframes at interactive speed. In order to validate our scheme we compared computer generated motions with motions performed by a sample of ten humans. These motions were defined by start and final postures and by an obstacle which had to be passed. In each case the generated positions are the same as those chosen by 30% of real humans, we therefore consider our method provides realistic motions. Moreover, the collision-free paths are automatically generated in a few seconds. Hence, our method can be very beneficial to animators by reducing the level of detail needed to define motions of articulated figures. It can also be used for the automatic generation of realistic animations for virtual reality applications.     

Interpolation synthesis of articulated figure motion

Most conventional media depend on engaging and appealing characters. Empty spaces and buildings would not fare well as television or movie programming, yet virtual reality usually offers up such spaces. The problem lies in the difficulty of creating computer generated characters that display real time, engaging interaction and realistic motion. Articulated figure motion for real time computer graphics offers one solution to this problem. A common approach stores a set of motions and lets you choose one particular motion at a time. The article describes a process that greatly expands the range of possible motions. Mixing motions selected from a database lets you create a new motion to exact specifications. The synthesized motion retains the original motions' subtle qualities, such as the realism of motion capture or the expressive, exaggerated qualities of artistic animation. Our method provides a new way to achieve inverse kinematics capability-for example, placing the hands or feet of an articulated figure in specific positions. It proves useful for both real time graphics and prerendered animation production. The method, called interpolation synthesis, is based on motion capture data and it provides real time character motion for interactive entertainment or avatars in virtual worlds     

基于多自主智能体的群体动画创作

群体动画一直是计算机动画界一个具有挑战性的研究方向，提出了一个基于多自主智能体的群体动画创作框架：群体中的各角色作为自主智能体，能感知环境信息，产生意图，规划行为，最后通过运动系统产生运动来完成行为和实现意图，与传统的角色运动生成机理不同，首先采用运动捕获系统建立基本运动库，然后通过运动编辑技术对基本运动进行处理以最终得到角色运动，应用本技术，动画师只需“拍摄”角色群体的运动就能创作群体动画，极大地提高了制作效率。     

基于非线性流形学习的3维人体运动合成

为了实现3维人体运动的有效合成,提出了一种基于非线性流形学习的3维人体运动合成框架及算法,并可应用于方便、快捷、用户可控的3维人体运动合成。该合成算法框架先采用非线性流形降维方法将高维运动样本映射到低维流形上,同时求解其本征运动语义参数空间的表达,然后将用户在低维运动语义参数空间中交互生成的样本通过逆向映射重建得到具有新运动语义特征的3维运动序列。实验结果表明该方法不仅能够对运动物理参数（如特定关节的运动位置、物理运动特征）进行较为精确的控制,还可用于合成具有高层运动语义（运动风格）的新运动数据。与现有运动合成方法比较,该方法具有用户可控、交互性强等优点,能够应用于常见3维人体运动数据的高效生成。     

多模态人机交互综述

多模态人机交互旨在利用语音、图像、文本、眼动和触觉等多模态信息进行人与计算机之间的信息交换。在生理心理评估、办公教育、军事仿真和医疗康复等领域具有十分广阔的应用前景。本文系统地综述了多模态人机交互的发展现状和新兴方向,深入梳理了大数据可视化交互、基于声场感知的交互、混合现实实物交互、可穿戴交互和人机对话交互的研究进展以及国内外研究进展比较。本文认为拓展新的交互方式、设计高效的各模态交互组合、构建小型化交互设备、跨设备分布式交互、提升开放环境下交互算法的鲁棒性等是多模态人机交互的未来研究趋势。     

Humanoid Audio–Visual Avatar With Emotive Text-to-Speech Synthesis

Emotive audio–visual avatars are virtual computer agents which have the potential of improving the quality of human-machine interaction and human-human communication significantly. However, the understanding of human communication has not yet advanced to the point where it is possible to make realistic avatars that demonstrate interactions with natural- sounding emotive speech and realistic-looking emotional facial expressions. In this paper, We propose the various technical approaches of a novel multimodal framework leading to a text-driven emotive audio–visual avatar. Our primary work is focused on emotive speech synthesis, realistic emotional facial expression animation, and the co-articulation between speech gestures (i.e., lip movements) and facial expressions. A general framework of emotive text-to-speech (TTS) synthesis using a diphone synthesizer is designed and integrated into a generic 3-D avatar face model. Under the guidance of this framework, we therefore developed a realistic 3-D avatar prototype. A rule-based emotive TTS synthesis system module based on the Festival-MBROLA architecture has been designed to demonstrate the effectiveness of the framework design. Subjective listening experiments were carried out to evaluate the expressiveness of the synthetic talking avatar.      

Real-time animation of realistic virtual humans

The authors have been working on simulating virtual humans for several years. Until recently, these constructs could not act in real time. Today, however, many applications need to simulate in real time virtual humans that look realistic. They have invested considerable effort in developing and integrating several modules into a system capable of animating humans in real-time situations. This includes interactive modules for building realistic individuals and a texture-fitting method suitable for all parts of the head and body. Animating the body, including the hands and their deformations, is the key aspect of the system; to their knowledge, no competing system integrates all these functions. They also included facial animation, as demonstrated below with virtual tennis players. They have developed a single system containing all the modules needed for simulating real-time virtual humans in distant virtual environments (VEs). The system lets one rapidly clone any individual and animate the clone in various contexts. People cannot mistake virtual humans for real ones, but they think them recognizable and realistic     

Real-time 3D human capture system for mixed-reality art and entertainment

A real-time system for capturing humans in 3D and placing them into a mixed reality environment is presented in this paper. Nine cameras surrounding her capture the subject. Looking through a head-mounted-display with a camera in front pointing at a marker, the user can see the 3D image of this subject overlaid onto a mixed reality scene. The 3D images of the subject viewed from this viewpoint are constructed using a robust and fast shape-from-silhouette algorithm. The paper also presents several techniques to produce good quality and speed up the whole system. The frame rate of our system is around 25 fps using only standard Intel processor-based personal computers. Besides a remote live 3D conferencing and collaborating system, we also describe an application of the system in art and entertainment, named Magic Land, which is a mixed reality environment where captured avatars of human and 3D computer generated virtual animations can form an interactive story and play with each other. This system demonstrates many technologies in human computer interaction: mixed reality, tangible interaction, and 3D communication. The result of the user study not only emphasizes the benefits, but also addresses some issues of these technologies.     

State of the Art in Hand and Finger Modeling and Animation

The human hand is a complex biological system able to perform numerous tasks with impressive accuracy and dexterity. Gestures furthermore play an important role in our daily interactions, and humans are particularly skilled at perceiving and interpreting detailed signals in communications. Creating believable hand motions for virtual characters is an important and challenging task. Many new methods have been proposed in the Computer Graphics community within the last years, and significant progress has been made towards creating convincing, detailed hand and finger motions. This state of the art report presents a review of the research in the area of hand and finger modeling and animation. Starting with the biological structure of the hand and its implications for how the hand moves, we discuss current methods in motion capturing hands, data‐driven and physics‐based algorithms to synthesize their motions, and techniques to make the appearance of the hand model surface more realistic. We then focus on areas in which detailed hand motions are crucial such as manipulation and communication. Our report concludes by describing emerging trends and applications for virtual hand animation.     

一种视频驱动的三维虚拟人动画实现方法

研究实现三维人体动画具有广泛的应用前景和实用意义,提出了一种二维视频驱动的三维人体动画实现方法。基于动态帧的关键帧提取算法从二维视频中构建了二维关键帧集合;基于二维关键帧构建二维人体骨骼模型;利用小孔成像原理和勾股定理计算得到关节特征点的深度坐标,从而得到了反映人体动画的三维数据。实验结果表明,该方法生成的三维人体动画形象逼真、成本低、提高了运动生成的实时性,能够应用于虚拟现实、计算机游戏、三维视频游戏制作等领域。     

Multimodal behavior realization for embodied conversational agents

Applications with intelligent conversational virtual humans, called Embodied Conversational Agents (ECAs), seek to bring human-like abilities into machines and establish natural human-computer interaction. In this paper we discuss realization of ECA multimodal behaviors which include speech and nonverbal behaviors. We devise RealActor, an open-source, multi-platform animation system for real-time multimodal behavior realization for ECAs. The system employs a novel solution for synchronizing gestures and speech using neural networks. It also employs an adaptive face animation model based on Facial Action Coding System (FACS) to synthesize face expressions. Our aim is to provide a generic animation system which can help researchers create believable and expressive ECAs.     

虚拟海洋场景的实时模拟研究与实现

为了提高虚拟海洋场景的实时性和真实感,提出了用多重Perlin噪声的叠加和波浪粒子相结合的方法建立实时海洋动画系统.该系统采用Perlin噪声和投影网格绘制技术模拟海面几何形态;并采用Perlin噪声和波浪粒子来跟踪水波的运动,从而快速、简便地实现了水上物体运动产生的水波特效;在光影效果上加入纹理技术建立光照模型,生成了复杂的、真实感强的海洋场景.仿真实验结果表明,该方法适用于大多数流体的模拟,渲染的海洋场景具有很好的真实感和时效性.     

Evaluation of quality and personalisation of VR/AR/MR learning systems

ABSTRACT

The paper aims to analyse the problem of quality evaluation and personalisation of virtual reality/augmented reality/mixed reality (VR/AR/MR). First of all, systematic review of relevant scientific literature on the research topic was conducted. After that, findings of the systematic review concerning evaluation of quality and personalisation of VR/AR/MR learning environments are presented. The author’s VR/AR/MR learning systems/environments quality evaluation and personalisation framework is also presented in the paper. Evaluation of quality of VR/AR/MR platforms/environments should be based on (a) applying both expert-centred (top-down) and user-centred (bottom-up) quality evaluation methods and (b) separating ‘internal quality’ criteria, and ‘quality in use’ criteria in the set of quality criteria (model). Personalisation of VR/AR/MR platforms/environments should be based on learners’ models/profiles using students’ learning styles, intelligent technologies, and Semantic Web applications.     

Virtual human animation in natural language visualisation

Simulation motion of Virtual Reality (VR) objects and humans has experienced important developments in the last decade. However, realistic virtual human animation generation remains a major challenge, even if applications are numerous, from VR games to medical training. This paper proposes different methods for animating virtual humans, including blending simultaneous animations of various temporal relations with multiple animation channels, minimal visemes for lip synchronisation, and space sites of virtual human and 3D object models for object grasping and manipulation. We present our work in our natural language visualisation (animation) system, CONFUCIUS, and describe how the proposed approaches are employed in CONFUCIUS’ animation engine.     

Designing the User Interface for Multimodal Speech and Pen-Based Gesture Applications: State-of-the-Art Systems and Future Research Directions

The growing interest in multimodal interface design is inspired in large part by the goals of supporting more transparent, flexible, efficient, and powerfully expressive means of human-computer interaction than in the past. Multimodal interfaces are expected to support a wider range of diverse applications, be usable by a broader spectrum of the average population, and function more reliably under realistic and challenging usage conditions. In this article, we summarize the emerging architectural approaches for interpreting speech and pen-based gestural input in a robust manner-including early and late fusion approaches, and the new hybrid symbolic-statistical approach. We also describe a diverse collection of state-of-the-art multimodal systems that process users' spoken and gestural input. These applications range from map-based and virtual reality systems for engaging in simulations and training, to field medic systems for mobile use in noisy environments, to web-based transactions and standard text-editing applications that will reshape daily computing and have a significant commercial impact. To realize successful multimodal systems of the future, many key research challenges remain to be addressed. Among these challenges are the development of cognitive theories to guide multimodal system design, and the development of effective natural language processing, dialogue processing, and error-handling techniques. In addition, new multimodal systems will be needed that can function more robustly and adaptively, and with support for collaborative multiperson use. Before this new class of systems can proliferate, toolkits also will be needed to promote software development for both simulated and functioning systems.     

基于计算机动画的虚拟导游技术及其应用

基于虚拟现实技术的虚拟旅游正蓬勃发展,正如导游在旅游业的重要地位一样,虚拟导游也是虚拟旅游技术的重要组成部分。该文利用计算机人物建模和动画技术,实现了基于计算机动画的虚拟导游,有效地增强了虚拟旅游场景的趣味性和真实感。讨论了基于计算机动画的虚拟导游的关键技术,并将其应用于虚拟旅游系统,效果良好。     

虚拟拆卸中多方式交互技术的应用

通过分析虚拟环境下不同人-机交互方式的特点，结合对产品装配体进行人工拆卸的操作活动，提出面向虚拟拆卸的人-机-零件间的多方式交互操作方法，在虚拟环境下研究和利用手抓取操作、手势操作、语音命令和3D虚拟菜单等多种交互方式对产品零部件进行自然、直观的人工拆卸，人机结合进行产品的装配／拆卸序列规划。     

B-spline wavelet-based motion smoothing

A B-spline wavelet-based algorithm is proposed in this study to solve the motion smoothing problem, which has practical applications in the computer animation and virtual reality (VR) areas. The motion of a rigid body consists of translation and rotation. The former is described by a space curve in three-dimensional Euclidean space, while the latter is represented by a curve in the unit quaternion space. Due to the complexity of rendering realistic rigid-body motions in the computer, rigid-body motions are often sampled from the physical world motions. Yet, sampling processes introduce noise; excessive noise results in tremulous motions in a VR environment. The noise-embedded motion data are decomposed using multiresolution analysis and the noise is detected as components of small magnitude. By eliminating the small-magnitude components, a smooth representation of the motion data is achieved.