Design and application of real-time visual attention model for the exploration of 3D virtual environments

This paper studies the design and application of a novel visual attention model designed to compute user's gaze position automatically, i.e., without using a gaze-tracking system. The model we propose is specifically designed for real-time first-person exploration of 3D virtual environments. It is the first model adapted to this context which can compute in real time a continuous gaze point position instead of a set of 3D objects potentially observed by the user. To do so, contrary to previous models which use a mesh-based representation of visual objects, we introduce a representation based on surface-elements. Our model also simulates visual reflexes and the cognitive processes which take place in the brain such as the gaze behavior associated to first-person navigation in the virtual environment. Our visual attention model combines both bottom-up and top-down components to compute a continuous gaze point position on screen that hopefully matches the user's one. We conducted an experiment to study and compare the performance of our method with a state-of-the-art approach. Our results are found significantly better with sometimes more than 100 percent of accuracy gained. This suggests that computing a gaze point in a 3D virtual environment in real time is possible and is a valid approach, compared to object-based approaches. Finally, we expose different applications of our model when exploring virtual environments. We present different algorithms which can improve or adapt the visual feedback of virtual environments based on gaze information. We first propose a level-of-detail approach that heavily relies on multiple-texture sampling. We show that it is possible to use the gaze information of our visual attention model to increase visual quality where the user is looking, while maintaining a high-refresh rate. Second, we introduce the use of the visual attention model in three visual effects inspired by the human visual system namely: depth-of-field blur, camera- motions, and dynamic luminance. All these effects are computed based on the simulated gaze of the user, and are meant to improve user's sensations in future virtual reality applications.     

Navigation in 3D virtual environments: Effects of user experience and location-pointing navigation aids

In this paper, we describe the results of an experimental study whose objective was twofold: (1) comparing three navigation aids that help users perform wayfinding tasks in desktop virtual environments (VEs) by pointing out the location of objects or places; (2) evaluating the effects of user experience with 3D desktop VEs on their effectiveness with the considered navigation aids. In particular, we compared navigation performance (in terms of total time to complete an informed search task) of 48 users divided into two groups: subjects in one group had experience in navigating 3D VEs while subjects in the other group did not. The experiment comprised four conditions that differed for the navigation aid that was employed. The first and the second condition, respectively, exploited 3D and 2D arrows to point towards objects that users had to reach; in the third condition, a radar metaphor was employed to show the location of objects in the VE; the fourth condition was a control condition with no location-pointing navigation aid available. The search task was performed both in a VE representing an outdoor geographic area and in an abstract VE that did not resemble any familiar environment. For each VE, users were also asked to order the four conditions according to their preference. Results show that the navigation aid based on 3D arrows outperformed (both in terms of user performance and user preference) the others, except in the case when it was used by experienced users in the geographic VE. In that case, it was as effective as the others. Finally, in the geographic VE, experienced users took significantly less time than inexperienced users to perform the informed search, while in the abstract VE the difference was significant only in the control and the radar conditions. From a more general perspective, our study highlights the need to take into specific consideration user experience in navigating VEs when designing navigation aids and evaluating their effectiveness.     

Inclusion of third-person perspective in CAVE-like immersive 3D virtual reality role-playing games for social reciprocity training of children with an autism spectrum disorder

Universal Access in the Information Society - The present study aimed to improve the ability of children with autism to recognize emotions correctly. We used our third-person perspective...     

A brush device with visual and haptic feedback for virtual painting of 3D virtual objects

We have previously developed a mixed reality (MR) painting system with which a user could take a physical object in the real world and apply virtual paint to it. However, this system could not provide the sensation of painting on virtual objects in MR space. Therefore, we subsequently proposed and developed mechanisms that simulated the effect of touch and movement when a brush device was used to paint on a virtual canvas. In this paper, we use visual and haptic feedback to provide the sensation of painting on virtual three-dimensional objects using a new brush device called the MAI Painting Brush++. We evaluate and confirm its effectiveness through several user studies.     

Auditory and visual 3D virtual reality therapy for chronic subjective tinnitus: theoretical framework

It is estimated that ~10% of the adult population in developed countries is affected by subjective tinnitus. Physiopathology of subjective tinnitus remains incompletely explained. Nevertheless, subjective tinnitus is thought to result from hyperactivity and neuroplastic reorganization of cortical and subcortical networks following acoustic deafferentation induced by cochlear or auditory nerve damage. Involvement of both auditory and non-auditory central nervous pathways explains the conscious perception of tinnitus and also the potentially incapacitating discomfort experienced by some patients (sound hypersensitivity, sleep disorders, attention deficit, anxiety or depression). These clinical patterns are similar to those observed in chronic pain following amputation where conditioning techniques using virtual reality have been shown both to be theoretically interesting and effectively useful. This analogy led us to develop an innovative setup with dedicated auditory and visual 3D virtual reality environments in which unilateral subjective tinnitus sufferers are given the possibility to voluntarily manipulate an auditory and visual image of their tinnitus (tinnitus avatar). By doing so, the patients will be able to transfer their subjective auditory perception to the tinnitus avatar and to gain agency on this multimodal virtual percept they hear, see and spatially control. Repeated sessions of such virtual reality immersions are then supposed to contribute to tinnitus treatment by promoting cerebral plasticity. This paper describes the theoretical framework and setup adjustments required by this very first attempt to adapt virtual reality techniques to subjective tinnitus treatment. Therapeutic usefulness will be validated by a further controlled clinical trial.     

Real-time 3D motion capture by monocular vision and virtual rendering

Networked 3D virtual environments allow multiple users to interact over the Internet by means of avatars and to get some feeling of a virtual telepresence. However, avatar control may be tedious. 3D sensors for motion capture systems based on 3D sensors have reached the consumer market, but webcams remain more widespread and cheaper. This work aims at animating a user’s avatar by real-time motion capture using a personal computer and a plain webcam. In a classical model-based approach, we register a 3D articulated upper-body model onto video sequences and propose a number of heuristics to accelerate particle filtering while robustly tracking user motion. Describing the body pose using wrists 3D positions rather than joint angles allows efficient handling of depth ambiguities for probabilistic tracking. We demonstrate experimentally the robustness of our 3D body tracking by real-time monocular vision, even in the case of partial occlusions and motion in the depth direction.     

Adaptive extended Kalman filtering for visual motion estimation of 3D objects

An algorithm for the real-time estimation of the position and orientation of a moving object of known geometry is presented in this paper. An estimation algorithm is adopted where a discrete-time extended Kalman filter computes the object pose on the basis of visual measurements of the object features. The scheme takes advantage of the prediction capability of the extended Kalman filter for the pre-selection of the features to be extracted from the image at each sample time. To enhance the robustness of the algorithm with respect to measurement noise and modelling error, an adaptive version of the extended Kalman filter, customized for visual applications, is proposed. Experimental results on a fixed single-camera visual system are presented to test the performance and the feasibility of the proposed approach.     

Model-based estimation of 3D human motion

This paper presents the formulations and techniques that we have developed for the 3D model-based, motion estimation of human movement from multiple cameras. Our method is based on the spatio-temporal analysis of the subject's silhouette and it has the advantage that the subject does not have to wear markers or other devices. We present tracking results from experiments involving the recovery of complex motions in the presence of significant occlusion     

The skin deformation of a 3D virtual human

This paper presents a skin deformation algorithm for creating 3D characters or virtual human models. The algorithm can be applied to rigid deformation, joint dependent localized deformation, skeleton driven deformation, cross contour deformation, and free-form deformation (FFD). These deformations are computed and demonstrated with examples and the algorithm is applied to overcome the difficulties in mechanically simulating the motion of the human body by club-shape models. The techniques described in this article enables the reconstruction of dynamic human models that can be used in defining and representing the geometrical and kinematical characteristics of human motion.     

3DMAX在虚拟人穴位模型三维重建中的应用

经络学是中医理论的重要组成部分,但经络的物质基础尚无定论;现有虚拟人体技术反映了丰富的解剖学上的细节,但未能包括对经络和中医人体穴位的描述。针对这些问题,以中医针推理论为指导,以中国人体穴位真实标本素材为基础,用3DMAX技术建立了一个对于人体穴位定位及针刺方向描述的模型,描述了利用三维可视化方法建立的空间模型,以及组织成为层次型文本知识体系的解释模型,以反映详细的解剖学相关知识     

Design and comparative evaluation of Smoothed Pointing: A velocity-oriented remote pointing enhancement technique

The increasing use of remote pointing devices in various application domains is fostering the adoption of pointing enhancement techniques which are aimed at counterbalancing the shortcomings of desk-free interaction. This paper describes the strengths and weaknesses of existing methods for ray pointing facilitation, and presents a refinement of Smoothed Pointing, an auto-calibrating velocity-oriented precision enhancing technique. Furthermore, the paper discusses the results of a user study aimed at empirically investigating how velocity-oriented approaches perform in target acquisition and in trajectory-based interaction tasks, considering both laser-style and image–plane pointing modalities. The experiments, carried out in a low precision scenario in which a Wiimote was used both as a wand and a tracking system, show that Smoothed Pointing allows a significant decrease in the error rate and achieves the highest values of throughput in trajectory-based tasks. The results also indicate that the effectiveness of precision enhancing techniques is significantly affected by the pointing modality and the type of pointing task.     

FI3D: direct-touch interaction for the exploration of 3D scientific visualization spaces

We present the design and evaluation of FI3D, a direct-touch data exploration technique for 3D visualization spaces. The exploration of three-dimensional data is core to many tasks and domains involving scientific visualizations. Thus, effective data navigation techniques are essential to enable comprehension, understanding, and analysis of the information space. While evidence exists that touch can provide higher-bandwidth input, somesthetic information that is valuable when interacting with virtual worlds, and awareness when working in collaboration, scientific data exploration in 3D poses unique challenges to the development of effective data manipulations. We present a technique that provides touch interaction with 3D scientific data spaces in 7 DOF. This interaction does not require the presence of dedicated objects to constrain the mapping, a design decision important for many scientific datasets such as particle simulations in astronomy or physics. We report on an evaluation that compares the technique to conventional mouse-based interaction. Our results show that touch interaction is competitive in interaction speed for translation and integrated interaction, is easy to learn and use, and is preferred for exploration and wayfinding tasks. To further explore the applicability of our basic technique for other types of scientific visualizations we present a second case study, adjusting the interaction to the illustrative visualization of fiber tracts of the brain and the manipulation of cutting planes in this context.     

Temporal motion models for monocular and multiview 3D human body tracking

We explore an approach to 3D people tracking with learned motion models and deterministic optimization. The tracking problem is formulated as the minimization of a differentiable criterion whose differential structure is rich enough for optimization to be accomplished via hill-climbing. This avoids the computational expense of Monte Carlo methods, while yielding good results under challenging conditions. To demonstrate the generality of the approach we show that we can learn and track cyclic motions such as walking and running, as well as acyclic motions such as a golf swing. We also show results from both monocular and multi-camera tracking. Finally, we provide results with a motion model learned from multiple activities, and show how this models might be used for recognition.     

Action-specific motion prior for efficient Bayesian 3D human body tracking

In this paper, we aim to reconstruct the 3D motion parameters of a human body model from the known 2D positions of a reduced set of joints in the image plane. Towards this end, an action-specific motion model is trained from a database of real motion-captured performances, and used within a particle filtering framework as a priori knowledge on human motion. First, our dynamic model guides the particles according to similar situations previously learnt. Then, the state space is constrained so only feasible human postures are accepted as valid solutions at each time step. As a result, we are able to track the 3D configuration of the full human body from several cycles of walking motion sequences using only the 2D positions of a very reduced set of joints from lateral or frontal viewpoints.     

Reduced mental load in learning a motor visual task with virtual 3D method

Distance learning is expanding rapidly, fueled by the novel technologies for shared recorded teaching sessions on the Web. Here, we ask whether 3D stereoscopic (3DS) virtual learning environment teaching sessions are more compelling than typical two‐dimensional (2D) video sessions and whether this type of teaching results in superior learning. The research goal was to compare learning in 2 virtual learning scenarios—on 2D displays and with an identical 3DS scenario. Participants watched a 2D or 3DS video of an instructor demonstrating a box origami paper‐folding task. We compared participants' folding test scores and self‐assessment questionnaires of the teaching scenarios and calculated their cognitive load index (CLI) based on electroencephalogram measurements during the observation periods. Results showed a highly significant difference between participants' folding test scores, CLI, and self‐assessment questionnaire results in 2D compared to 3DS sessions. Our findings indicate that employing stereoscopic 3D technology over 2D displays in the design of emerging virtual and augmented reality applications in distance learning has advantages.     

双目视觉下三维人体运动跟踪算法*

由于人体运动的复杂性,人体运动轨迹的快速改变和人体自遮挡现象经常发生,这给人体运动跟踪带来了很大的困难。针对此问题提出了一种基于三维Kalman滤波器和人体约束的人体运动跟踪算法。该算法首先利用外极线约束和灰度互相关法对二维标记点进行立体匹配,计算各个标记点的三维位置,从而构建得到三维标记点;然后利用三维Kalman滤波器对三维标记点进行跟踪;最后利用人体约束检验和修正跟踪结果。实验结果表明,该算法能有效地对复杂人体动作进行跟踪并能从跟踪错误中正确恢复。     

A computational framework and an algorithm for the measurement of visual motion

The robust measurement of visual motion from digitized image sequences has been an important but difficult problem in computer vision. This paper describes a hierarchical computational framework for the determination of dense displacement fields from a pair of images, and an algorithm consistent with that framework. Our framework is based on a scale-based separation of the image intensity information and the process of measuring motion. The large-scale intensity information is first used to obtain rough estimates of image motion, which are then refined by using intensity information at smaller scales. The estimates are in the form of displacement (or velocity) vectors for pixels and are accompanied by a direction-dependent confidence measure. A smoothness constraint is employed to propagate measurements with high confidence to neighboring areas where the confidences are low. At all levels, the computations are pixel-parallel, uniform across the image, and based on information from a small neighborhood of a pixel. Results of applying our algorithm to pairs of real images are included. In addition to our own matching algorithm, we also show that two different hierarchical gradient-based algorithms are consistent with our framework.     

Automatic alignment for virtual fitting using 3D garment stretching and human body relocation

In recent years, researches on virtual fitting are mostly based on the data of garment pieces through Computer Aided Design. However, three-dimensional garment models are more intuitive, integrated and easy to access and manage. Therefore, we propose and design a method to automatically align virtual humans with three-dimensional garment models. Firstly, we basically align two models through three-dimensional geometric calculation and postural alteration of virtual humans. Then, we detect the penetration phenomena and define the points of the garment model penetrating into the human model as penetration points. Based on the penetration detection, we use Poisson mesh deformation to adjust the penetration points by imitating the adjustment process of dragging clothes. Finally, the results obtained from alignment can be directly used for physical-based simulation. Experimental results show that the method we propose can achieve good effects of alignment, and has good versatility and practicality.