3D Video Recorder: a System for Recording and Playing Free‐Viewpoint Video†

We present the 3D Video Recorder, a system capable of recording, processing, and playing three‐dimensional video from multiple points of view. We first record 2D video streams from several synchronized digital video cameras and store pre‐processed images to disk. An off‐line processing stage converts these images into a time‐varying 3D hierarchical point‐based data structure and stores this 3D video to disk. We show how we can trade‐off 3D video quality with processing performance and devise efficient compression and coding schemes for our novel 3D video representation. A typical sequence is encoded at less than 7 Mbps at a frame rate of 8.5 frames per second. The 3D video player decodes and renders 3D videos from hard‐disk in real‐time, providing interaction features known from common video cassette recorders, like variable‐speed forward and reverse, and slow motion. 3D video playback can be enhanced with novel 3D video effects such as freeze‐and‐rotate and arbitrary scaling. The player builds upon point‐based rendering techniques and is thus capable of rendering high‐quality images in real‐time. Finally, we demonstrate the 3D Video Recorder on multiple real‐life video sequences. ACM CSS: I.3.2 Computer Graphics—Graphics Systems, I.3.5 Computer Graphics—Computational Geometry and Object Modelling, I.3.7 Computer Graphics—Three‐Dimensional Graphics and Realism     

Generation of augmented video sequences combining behavioral animation and multi‐object tracking

In this paper we present a novel approach to generate augmented video sequences in real‐time, involving interactions between virtual and real agents in real scenarios. On the one hand, real agent motion is estimated by means of a multi‐object tracking algorithm, which determines real objects' position over the scenario for each time step. On the other hand, virtual agents are provided with behavior models considering their interaction with the environment and with other agents. The resulting framework allows to generate video sequences involving behavior‐based virtual agents that react to real agent behavior and has applications in education, simulation, and in the game and movie industries. We show the performance of the proposed approach in an indoor and outdoor scenario simulating human and vehicle agents. Copyright © 2009 John Wiley & Sons, Ltd.     

Visualizing Crowds in Real-Time

Real‐time crowd visualization has recently attracted quite an interest from the graphics community and, asinteractive applications become even more complex, there is a natural demand for new and unexplored applicationscenarios. However, the interactive simulation of complex environments populated by large numbers of virtualcharacters is a composite problem which poses serious difficulties even on modern computer hardware. In thispaper we look at methods to deal with various aspects of crowd visualization, ranging from collision detectionand behaviour modeling to fast rendering with shadows and quality shading. These methods make extensive useof current graphics hardware capabilities with the aim of providing scalability without compromising run‐timespeed. Results from a system employing these techniques seem to suggest that simulations of reasonably complexenvironments populated with thousands of animated characters are possible in real‐time. ACM CSS: I.3.7 Three‐Dimensional Graphics and Realism—Animation     

Urban Walkability Design Using Virtual Population Simulation

We present a system to generate a procedural environment that produces a desired crowd behaviour. Instead of altering the behavioural parameters of the crowd itself, we automatically alter the environment to yield such desired crowd behaviour. This novel inverse approach is useful both to crowd simulation in virtual environments and to urban crowd planning applications. Our approach tightly integrates and extends a space discretization crowd simulator with inverse procedural modelling. We extend crowd simulation by goal exploration (i.e. agents are initially unaware of the goal locations), variable‐appealing sign usage and several acceleration schemes. We use Markov chain Monte Carlo to quickly explore the solution space and yield interactive design. We have applied our method to a variety of virtual and real‐world locations, yielding one order of magnitude faster crowd simulation performance over related methods and several fold improvement of crowd indicators.     

Real‐Time Consensus‐Based Scene Reconstruction Using Commodity Graphics Hardware†

We present a novel use of commodity graphics hardware that effectively combines a plane‐sweeping algorithm with view synthesis for real‐time, online 3D scene acquisition and view synthesis. Using real‐time imagery from a few calibrated cameras, our method can generate new images from nearby viewpoints, estimate a dense depth map from the current viewpoint, or create a textured triangular mesh. We can do each of these without any prior geometric information or requiring any user interaction, in real time and online. The heart of our method is to use programmable Pixel Shader technology to square intensity differences between reference image pixels, and then to choose final colors (or depths) that correspond to the minimum difference, i.e. the most consistent color. In this paper we describe the method, place it in the context of related work in computer graphics and computer vision, and present some results. ACM CSS: I.3.3 Computer Graphics—Bitmap and framebuffer operations, I.4.8 Image Processing and Computer Vision—Depth cues, Stereo     

A Spoken Dialogue System for Navigation in Non-Immersive Virtual Environments

Navigation is the process by which people control their movement in virtual environments and is a corefunctional requirement for all virtual environment (VE) applications. Users require the ability to move, controllingorientation, direction of movement and speed, in order to achieve a particular goal within a VE. Navigation israrely the end point in itself (which is typically interaction with the visual representations of data) but applicationsoften place a high demand on navigation skills, which in turn means that a high level of support for navigationis required from the application. On desktop systems navigation in non‐immersive systems is usually supportedthrough the usual hardware devices of mouse and keyboard. Previous work by the authors shows that many usersexperience frustration when trying to perform even simple navigation tasks — users complain about getting lost,becoming disorientated and finding the interface `difficult to use'. In this paper we report on work in progressin exploiting natural language processing (NLP) technology to support navigation in non‐immersive virtualenvironments. A multi‐modal system has been developed which supports a range of high‐level (spoken) navigationcommands and indications are that spoken dialogue interaction is an effective alternative to mouse and keyboardinteraction for many tasks. We conclude that multi‐modal interaction, combining technologies such as NLP withmouse and keyboard may offer the most effective interaction with VEs and identify a number of areas where furtherwork is necessary. ACM CSS: I.3.6 Computer Graphics Methodology and Techniques—Interaction and Techniques, I.3.7 Three‐DimensionalGraphics and Realism—Virtual Reality, I.2.7 Natural Language Processing—Speech Recognitionand Synthesis     

Recent Developments and Applications of Haptic Devices

Over recent years a variety of haptic feedback devices have been developed and are being used in a number of important applications. They range from joysticks used in the entertainment industry to specialised devices used in medical applications. This paper will describe the recent developments of these devices and show how they have been applied. It also examines how haptic feedback has been combined with visual display devices, such as virtual reality walls and workbenches, in order to improve the immersive experience. ACM CSS: H.5.2 Information Interfaces and Presentation—Haptic I/O; I.3.8 Computer Graphics—Applications; I.6 Simulation and Modelling—Applications     

Power‐aware provisioning of virtual machines for real‐time Cloud services

Reducing power consumption has been an essential requirement for Cloud resource providers not only to decrease operating costs, but also to improve the system reliability. As Cloud computing becomes emergent for the Anything as a Service (XaaS) paradigm, modern real‐time services also become available through Cloud computing. In this work, we investigate power‐aware provisioning of virtual machines for real‐time services. Our approach is (i) to model a real‐time service as a real‐time virtual machine request; and (ii) to provision virtual machines in Cloud data centers using dynamic voltage frequency scaling schemes. We propose several schemes to reduce power consumption by hard real‐time services and power‐aware profitable provisioning of soft real‐time services. Copyright © 2011 John Wiley & Sons, Ltd.     

Position‐Based Skinning for Soft Articulated Characters

In this paper, we introduce a two‐layered approach addressing the problem of creating believable mesh‐based skin deformation. For each frame, the skin is first deformed with a classic linear blend skinning approach, which usually leads to unsightly artefacts such as the well‐known candy‐wrapper effect and volume loss. Then we enforce some geometric constraints which displace the positions of the vertices to mimic the behaviour of the skin and achieve effects like volume preservation and jiggling. We allow the artist to control the amount of jiggling and the area of the skin affected by it. The geometric constraints are solved using a position‐based dynamics (PBDs) schema. We employ a graph colouring algorithm for parallelizing the computation of the constraints. Being based on PBDs guarantees efficiency and real‐time performances while enduring robustness and unconditional stability. We demonstrate the visual quality and the performance of our approach with a variety of skeleton‐driven soft body characters.     

Using a Visual Attention Model to Improve Gaze Tracking Systems in Interactive 3D Applications

This paper introduces the use of a visual attention model to improve the accuracy of gaze tracking systems. Visual attention models simulate the selective attention part of the human visual system. For instance, in a bottom‐up approach, a saliency map is defined for the image and gives an attention weight to every pixel of the image as a function of its colour, edge or intensity. Our algorithm uses an uncertainty window, defined by the gaze tracker accuracy, and located around the gaze point given by the tracker. Then, using a visual attention model, it searches for the most salient points, or objects, located inside this uncertainty window, and determines a novel, and hopefully, better gaze point. This combination of a gaze tracker together with a visual attention model is considered as the main contribution of the paper. We demonstrate the promising results of our method by presenting two experiments conducted in two different contexts: (1) a free exploration of a visually rich 3D virtual environment without a specific task, and (2) a video game based on gaze tracking involving a selection task. Our approach can be used to improve real‐time gaze tracking systems in many interactive 3D applications such as video games or virtual reality applications. The use of a visual attention model can be adapted to any gaze tracker and the visual attention model can also be adapted to the application in which it is used.     

Stabilization of electromechanical systems via interconnection and damping assignment

In this article, we propose a general controller structure for asymptotic position regulation of electromechanical systems derived using the Interconnection and Damping Assignment Passivity‐Based Control methodology recently proposed in the literature. The controller is applicable to arbitrary fully actuated electromechanical systems with linear magnetic materials consisting of inductances, permanent magnets, and one mechanical co‐ordinate. We assume linear magnetic materials and fully actuated electrical dynamics; however, no restrictions are imposed on the particular form of the parameters that define the system dynamics, i.e. the inductance matrix, the magnetic coupling or the potential energy function. This allows us to treat—in a unified framework and without any additional simplifying assumptions—very diverse applications, including magnetic suspensions, and stepper and permanent magnet synchronous motors. Instrumental for our developments is the inclusion of ‘virtual’ couplings between the electrical and the mechanical subsystem, which is naturally suggested in this control methodology. Copyright © 2003 John Wiley & Sons, Ltd.     

Towards video-based immersive environments

Video provides a comprehensive visual record of environment activity over time. Thus, video data is an attractive source of information for the creation of virtual worlds which require some real-world fidelity. This paper describes the use of multiple streams of video data for the creation of immersive virtual environments. We outline our multiple perspective interactive video (MPI-Video) architecture which provides the infrastructure for the processing and analysis of multiple streams of video data. Our MPI-Video system performs automated analysis of the raw video and constructs a model of the environment and object activity within this environment. This model provides a comprehensive representation of the world monitored by the cameras which, in turn, can be used in the construction of a virtual world. In addition, using the information produced and maintained by the MPI-Video system, our immersive video system generates virtual video sequences. These are sequences of the dynamic environment from an arbitrary view point generated using the real camera data. Such sequences allow a user to navigate through the environment and provide a sense of immersion in the scene. We discuss results from our MPI-Video prototype, outline algorithms for the construction of virtual views and provide examples of a variety of such immersive video sequences.     

An interactional view of social presence: Making the virtual other “real”

In IS research, social presence is generally defined as the perceived capacity of a communication medium to convey contextual cues normally available in face‐to‐face settings. However, theorizing social presence as a property of the technology has been challenged for decades. The objective of this paper is to develop a more contemporary, interactional view of social presence. To this end, this paper develops a new conceptualization of how participants form the sense that each other is present. We characterize the development of this sense as a skilful accomplishment that entails interactants' joint construction of each other as “real.” Viewing social presence as contingent on social practice, we seek to answer the following research question: “How is social presence accomplished in virtual environments?” To explicate how virtual others are perceived as becoming socially present, that is, emotionally and psychologically “real” to someone interacting with them, we draw from Goffman's work, particularly his concepts of involvement and involvement obligation. Detailing two examples of social interaction in the virtual world Second Life, our analysis highlights the key role that this moral obligation, intrinsic to everyday social interaction, plays in virtual others becoming perceived as psychoemotionally “there.” By outlining a model of how the sense of a virtual other as “real” is produced in and through social interaction, our work contributes a sociological perspective to the construct of social presence and underscores some of the material and social conditions necessary for users to perceive virtual others as present.     

Accurate garbage collection in uncooperative environments revisited

Implementing a concurrent programming language such as Java by means of a translator to an existing language is attractive as it provides portability over all platforms supported by the host language and reduces development time—as many low‐level tasks can be delegated to the host compiler. The C and C++ programming languages are popular choices for many language implementations due to the availability of efficient compilers on a wide range of platforms. For garbage‐collected languages, however, they are not a perfect match as no support is provided for accurately discovering pointers to heap‐allocated data on thread stacks. We evaluate several previously published techniques and propose a new mechanism, lazy pointer stacks, for performing accurate garbage collection in such uncooperative environments. We implemented the new technique in the Ovm Java virtual machine with our own Java‐to‐C/C++ compiler using GCC as a back‐end compiler. Our extensive experimental results confirm that lazy pointer stacks outperform existing approaches: we provide a speedup of 4.5% over Henderson's accurate collector with a 17% increase in code size. Accurate collection is essential in the context of real‐time systems, we thus validate our approach with the implementation of a real‐time concurrent garbage collection algorithm. Copyright © 2009 John Wiley & Sons, Ltd.     

Image Interpolation by Pixel-Level Data-Dependent Triangulation

We present a novel image interpolation algorithm. The algorithm can be used in arbitrary resolution enhancement, arbitrary rotation and other applications of still images in continuous space. High‐resolution images are interpolated from the pixel‐level data‐dependent triangulation of lower‐resolution images. It is simpler than other methods and is adaptable to a variety of image manipulations. Experimental results show that the new “mesh image” algorithm is as fast as the bilinear interpolation method. We assess the interpolated images' quality visually and also by the MSE measure which shows our method generates results comparable in quality to slower established methods. We also implement our method in graphics card hardware using OpenGL which leads to real‐time high‐quality image reconstruction. These features give it the potential to be used in gaming and image‐processing applications.     

Simulated time for host‐based testing with TTCN‐3

Prior to testing embedded software in a target environment, it is usually tested in a host environment used for developing the software. When a system is tested in a host environment, its real‐time behaviour is affected by the use of simulators, emulation and monitoring. In this paper, the authors provide a semantics for host‐based testing with simulated time and propose a simulated‐time solution for distributed testing with TTCN‐3, which is a standardized language for specifying and executing test suites. The paper also presents the application of testing with simulated time to two real‐life systems. Copyright © 2007 John Wiley & Sons, Ltd.     

Non‐fragile H ∞ control for switched stochastic delay systems with application to water quality process

In this paper, the problem of non‐fragile observer‐based H _∞ control for discrete‐time switched delay systems is investigated. Both data missing and time delays are taken into account in the links from sensors to observers and from controllers to actuators. Because data missing satisfies the Bernoulli distribution, such problem is transformed into an H _∞ control problem for stochastic switched delay systems. Average dwell time approach is used to obtain sufficient conditions on the solvability of such problems. A numerical example and a real example for water quality control are provided to illustrate the effectiveness and potential applications of the proposed techniques. Copyright © 2013 John Wiley & Sons, Ltd.     

Designing and implementing lightweight kernels for capability computing

In the early 1990s, researchers at Sandia National Laboratories and the University of New Mexico began development of customized system software for massively parallel ‘capability’ computing platforms. These lightweight kernels have proven to be essential for delivering the full power of the underlying hardware to applications. This claim is underscored by the success of several supercomputers, including the Intel Paragon, Intel Accelerated Strategic Computing Initiative Red, and the Cray XT series of systems, each having established a new standard for high‐performance computing upon introduction. In this paper, we describe our approach to lightweight compute node kernel design and discuss the design principles that have guided several generations of implementation and deployment. A broad strategy of operating system specialization has led to a focus on user‐level resource management, deterministic behavior, and scalable system services. The relative importance of each of these areas has changed over the years in response to changes in applications and hardware and system architecture. We detail our approach and the associated principles, describe how our application of these principles has changed over time, and provide design and performance comparisons to contemporaneous supercomputing operating systems. Copyright © 2008 John Wiley & Sons, Ltd.     

Modelling selective visual attention for autonomous virtual characters

Autonomous virtual characters (AVCs) are becoming more prevalent both for real‐time interaction and also as digital actors in film and TV production. AVCs require believable virtual human animations, accompanied by natural attention generation, and thus the software that controls the AVCs needs to model when and how to interact with the objects and other characters that exist in the virtual environment. This paper models automatic attention behaviour using a saliency model that generates plausible targets for combined gaze and head motions. The model was compared with the default behaviour of the Second Life (SL) system in an object observation scenario while it was compared with real actors' behaviour in a conversation scenario. Results from a study run within the SL system demonstrate a promising attention model that is not just believable and realistic but also adaptable to varying task, without any prior knowledge of the virtual scene. Copyright © 2011 John Wiley & Sons, Ltd.