Advances in Interaction with 3D Environments

Various interaction techniques have been developed for interactive 3D environments. This paper presents an up‐to‐date and comprehensive review of the state of the art of non‐immersive interaction techniques for Navigation, Selection & Manipulation, and System Control, including a basic introduction to the topic, the challenges and an examination of a number of popular approaches. We also introduce 3D Interaction Testbed (3DIT) to firstly allow a ‘hands‐on' understanding of 3D interaction principles, and secondly to create an open platform for defining evaluation methods, stimuli as well as representative tasks akin to those found in other disciplines of science. We hope that this survey can aid both researchers and developers of interactive 3D applications in having a clearer overview of the topic and in particular can be useful for practitioners and researchers that are new to the field of interactive 3D graphics.     

Real‐Time Fluid Effects on Surfaces using the Closest Point Method

The Closest Point Method (CPM) is a method for numerically solving partial differential equations (PDEs) on arbitrary surfaces, independent of the existence of a surface parametrization. The CPM uses a closest point representation of the surface, to solve the unmodified Cartesian version of a surface PDE in a 3D volume embedding, using simple and well‐understood techniques. In this paper, we present the numerical solution of the wave equation and the incompressible Navier‐Stokes equations on surfaces via the CPM, and we demonstrate surface appearance and shape variations in real‐time using this method. To fully exploit the potential of the CPM, we present a novel GPU realization of the entire CPM pipeline. We propose a surface‐embedding adaptive 3D spatial grid for efficient representation of the surface, and present a high‐performance approach using CUDA for converting surfaces given by triangulations into this representation. For real‐time performance, CUDA is also used for the numerical procedures of the CPM. For rendering the surface (and the PDE solution) directly from the closest point representation without the need to reconstruct a triangulated surface, we present a GPU ray‐casting method that works on the adaptive 3D grid.     

Learning 3D Deformation of Animals from 2D Images

Understanding how an animal can deform and articulate is essential for a realistic modification of its 3D model. In this paper, we show that such information can be learned from user‐clicked 2D images and a template 3D model of the target animal. We present a volumetric deformation framework that produces a set of new 3D models by deforming a template 3D model according to a set of user‐clicked images. Our framework is based on a novel locally‐bounded deformation energy, where every local region has its own stiffness value that bounds how much distortion is allowed at that location. We jointly learn the local stiffness bounds as we deform the template 3D mesh to match each user‐clicked image. We show that this seemingly complex task can be solved as a sequence of convex optimization problems. We demonstrate the effectiveness of our approach on cats and horses, which are highly deformable and articulated animals. Our framework produces new 3D models of animals that are significantly more plausible than methods without learned stiffness.     

Sets of Globally Optimal Stream Surfaces for Flow Visualization

Stream surfaces are a well‐studied and widely used tool for the visualization of 3D flow fields. Usually, stream surface seeding is carried out manually in time‐consuming trial and error procedures. Only recently automatic selection methods were proposed. Local methods support the selection of a set of stream surfaces, but, contrary to global selection methods, they evaluate only the quality of the seeding lines but not the quality of the whole stream surfaces. Global methods, on the other hand, only support the selection of a single optimal stream surface until now. However, for certain flow fields a single stream surface is not sufficient to represent all flow features. In our work, we overcome this limitation by introducing a global selection technique for a set of stream surfaces. All selected surfaces optimize global stream surface quality measures and are guaranteed to be mutually distant, such that they can convey different flow features. Our approach is an efficient extension of the most recent global selection method for single stream surfaces. We illustrate its effectiveness on a number of analytical and simulated flow fields and analyze the quality of the results in a user study.     

A Sketch-Based User Interface for Reconstructing Architectural Drawings

We present a framework for interactive sketching that allows users to create three‐dimensional (3D) architectural models quickly and easily from a source drawing. The sketching process has four steps. (1) The user calibrates a viewing camera by specifying the origin and vanishing points of the drawing. (2) The user outlines surface polygons in the drawing. (3) A 3D reconstruction algorithm uses perceptual constraints to determine the closest visual fit for the polygon. (4) The user can then adjust aesthetic controls to produce several stylistic effects in the scene: a smooth transition between day and night rendering, a horizon knockout effect and entourage figures. The major advantage of our approach lies in the combination of perception‐based techniques, which allow us to minimize unnecessary interactions, and a hinging‐angle scheme, which shows significant improvement in numerical stability over previous optimization‐based 3D reconstruction algorithms. We also demonstrate how our reconstruction algorithm can be extended to work with perspective images, a feature unavailable in previous approaches.     

CubeCover– Parameterization of 3D Volumes

Despite the success of quad‐based 2D surface parameterization methods, effective parameterization algorithms for 3D volumes with cubes, i.e. hexahedral elements, are still missing. Cube Cover is a first approach for generating a hexahedral tessellation of a given volume with boundary aligned cubes which are guided by a frame field. The input of Cube Cover is a tetrahedral volume mesh. First, a frame field is designed with manual input from the designer. It guides the interior and boundary layout of the parameterization. Then, the parameterization and the hexahedral mesh are computed so as to align with the given frame field. Cube Cover has similarities to the Quad Cover algorithm and extends it from 2D surfaces to 3D volumes. The paper also provides theoretical results for 3D hexahedral parameterizations and analyses topological properties of the appropriate function space.     

A Linear Variational System for Modelling From Curves

We present a linear system for modelling 3D surfaces from curves. Our system offers better performance, stability and precision in control than previous non‐linear systems. By exploring the direct relationship between a standard higher‐order Laplacian editing framework and Hermite spline curves, we introduce a new form of Cauchy constraint that makes our system easy to both implement and control. We introduce novel workflows that simplify the construction of 3D models from sketches. We show how to convert existing 3D meshes into our curve‐based representation for subsequent editing and modelling, allowing our technique to be applied to a wide range of existing 3D content.     

Semi‐Supervised Learning in Reconstructed Manifold Space for 3D Caricature Generation

Recently, automatic 3D caricature generation has attracted much attention from both the research community and the game industry. Machine learning has been proven effective in the automatic generation of caricatures. However, the lack of 3D caricature samples makes it challenging to train a good model. This paper addresses this problem by two steps. First, the training set is enlarged by reconstructing 3D caricatures. We reconstruct 3D caricatures based on some 2D caricature samples with a Principal Component Analysis (PCA)‐based method. Secondly, between the 2D real faces and the enlarged 3D caricatures, a regressive model is learnt by the semi‐supervised manifold regularization (MR) method. We then predict 3D caricatures for 2D real faces with the learnt model. The experiments show that our novel approach synthesizes the 3D caricature more effectively than traditional methods. Moreover, our system has been applied successfully in a massive multi‐user educational game to provide human‐like avatars.     

Streaming 3D deforming surfaces with dynamic resolution control

Real‐time streaming of shape deformations in a shared distributed virtual environment is a challenging task due to the difficulty of transmitting large amounts of 3D animation data to multiple receiving parties at a high frame rate. In this paper, we present a framework for streaming 3D shape deformations, which allows shapes with multi‐resolutions to share the same deformations simultaneously in real time. The geometry and motion of deforming mesh or point‐sampled surfaces are compactly encoded, transmitted, and reconstructed using the spectra of the manifold harmonics. A receiver‐based multi‐resolution surface reconstruction approach is introduced, which allows deforming shapes to switch smoothly between continuous multi‐resolutions. On the basis of this dynamic reconstruction scheme, a frame rate control algorithm is further proposed to achieve rendering at interactive rates. We also demonstrate an efficient interpolation‐based strategy to reduce computing of deformation. The experiments conducted on both mesh and point‐sampled surfaces show that our approach achieves efficient performance even if deformations of complex 3D surfaces are streamed. Copyright © 2013 John Wiley & Sons, Ltd.     

GeoBrush: Interactive Mesh Geometry Cloning

We propose a method for interactive cloning of 3D surface geometry using a paintbrush interface, similar to the continuous cloning brush popular in image editing. Existing interactive mesh composition tools focus on atomic copy‐and‐paste of preselected feature areas, and are either limited to copying surface displacements, or require the solution of variational optimization problems, which is too expensive for an interactive brush interface. In contrast, our GeoBrush method supports real‐time continuous copying of arbitrary high‐resolution surface features between irregular meshes, including topological handles. We achieve this by first establishing a correspondence between the source and target geometries using a novel generalized discrete exponential map parameterization. Next we roughly align the source geometry with the target shape using Green Coordinates with automatically‐constructed cages. Finally, we compute an offset membrane to smoothly blend the pasted patch with C continuity before stitching it into the target. The offset membrane is a solution of a bi‐harmonic PDE, which is computed on the GPU in real time by exploiting the regular parametric domain. We demonstrate the effectiveness of GeoBrush with various editing scenarios, including detail enrichment and completion of scanned surfaces.     

Diorama Construction From a Single Image

Diorama artists produce a spectacular 3D effect in a confined space by generating depth illusions that are faithful to the ordering of the objects in a large real or imaginary scene. Indeed, cognitive scientists have discovered that depth perception is mostly affected by depth order and precedence among objects. Motivated by these findings, we employ ordinal cues to construct a model from a single image that similarly to Dioramas, intensifies the depth perception. We demonstrate that such models are sufficient for the creation of realistic 3D visual experiences. The initial step of our technique extracts several relative depth cues that are well known to exist in the human visual system. Next, we integrate the resulting cues to create a coherent surface. We introduce wide slits in the surface, thus generalizing the concept of cardboard cutout layers. Lastly, the surface geometry and texture are extended alongside the slits, to allow small changes in the viewpoint which enriches the depth illusion.     

The effect of stereoscopic viewing in a word‐search task with a layered background

Abstract— The benefits of stereoscopic viewing were explored in searching in words superimposed over a background. In the first experiment, eight participants searched for text in a normal 2‐D display, a 3‐D display using a parallax barrier, and a darkened 2‐D display of equivalent brightness to the 3‐D display. Word‐search performance was significantly faster for the bright 2‐D display vs. the 3‐D display, but when brightness was controlled, performance on the 3‐D display was better relative to the 2‐D (dim) display. In a second experiment, the effect of floating text vs. sinking background disparity was assessed across four background conditions. Twenty participants saw only the floating‐text (FT) condition and 20 participants saw only the sinking‐background (SB) condition. Performance of the SB group was significantly better than that of FT group, and the advantage of SB disparity was greater with the more‐complex backgrounds. Thus, when a parallax‐barrier 3‐D display is used to view text or other figural information overlaid on a background, it is proposed that the layer of primary interest (foreground) should be displayed with zero disparity (on the physical display surface) with the secondary layer (background) appearing to be sunk beneath that surface.     

A wide view auto‐stereoscopic 3D display with an eye‐tracking system for enhanced horizontal viewing position and viewing distance

This paper describes the development of auto‐stereoscopic three‐dimensional (3D) display with an eye‐tracking system for not only the X‐axis (right–left) and Y‐axis (up–down) plane directions but also the Z‐axis (forward–backward) direction. In the past, the eye‐tracking 3D system for the XY‐axes plane directions that we had developed had a narrow 3D viewing space in the Z‐axis direction because of occurrence of 3D crosstalk variation on screen. The 3D crosstalk variation on screen was occurred when the viewer's eye position moved back and forth along the Z‐axis direction. The reason was that the liquid crystal (LC) barrier pitch was fixed and the LC barrier was able to control the only barrier aperture position. To solve this problem, we developed the LC barrier that is able to control the barrier pitch as well as the barrier aperture position in real time, corresponding to the viewer's eye position. As a result, the 3D viewing space has achieved to expand up to 320–1016 mm from the display surface in the Z‐axis direction and within a range of ±267 mm in the X‐axis direction. In terms of the Y‐axis direction, the viewing space is not necessary to be considered, because of a stripe‐shaped parallax barrier.     

Reconstructing Animated Meshes from Time‐Varying Point Clouds

In this paper, we describe a novel approach for the reconstruction of animated meshes from a series of time‐deforming point clouds. Given a set of unordered point clouds that have been captured by a fast 3‐D scanner, our algorithm is able to compute coherent meshes which approximate the input data at arbitrary time instances. Our method is based on the computation of an implicit function in ?⁴ that approximates the time‐space surface of the time‐varying point cloud. We then use the four‐dimensional implicit function to reconstruct a polygonal model for the first time‐step. By sliding this template mesh along the time‐space surface in an as‐rigid‐as‐possible manner, we obtain reconstructions for further time‐steps which have the same connectivity as the previously extracted mesh while recovering rigid motion exactly. The resulting animated meshes allow accurate motion tracking of arbitrary points and are well suited for animation compression. We demonstrate the qualities of the proposed method by applying it to several data sets acquired by real‐time 3‐D scanners.     

Autostereoscopic Rendering of Multiple Light Fields

Light fields were introduced a decade ago as a new high‐dimensional graphics rendering model. However, they have not been thoroughly used because their applications are very specific and their storage requirements are too high. Recently, spatial imaging devices have been related to light fields. These devices allow several users to see three‐dimensional (3D) images without using glasses or other intrusive elements. This paper presents a light‐field model that can be rendered in an autostereoscopic spatial device. The model is viewpoint‐independent and supports continuous multiresolution, foveal rendering, and integrating multiple light fields and geometric models in the same scene. We also show that it is possible to examine interactively a scene composed of several light fields and geometric models. Visibility is taken care of by the algorithm. Our goal is to apply our models to 3D TV and spatial imaging.     

Representation of Pseudo Inter-reflection and Transparency by Considering Characteristics of Human Vision

We have succeeded in developing a quick and fully automated system that can generate photo‐realistic 3D CG data based on a real object. A major factor in this success comes from our findings through psychophysical experiments that human observers do not have an accurate idea of what should be actually reflected as inter‐reflections on the surface of an object. Taking advantage of this characteristic of human vision, we propose a new inter‐reflection representation technique in which inter‐reflections are simulated by allowing the same quantity of reflection components as there are in the background to pass through the object. Since inter‐reflection and transparency are calculated by the same algorithm, our system can capture 3D CG data from various real objects having a strong inter‐reflection, such as plastic and porcelain items or translucent glass and acrylic resin objects. The synthetic images from the 3D CG data generated with this pseudo inter‐reflection and transparency look very natural. In addition, the 3D CG data and synthetic images are produced quickly at a lower cost.     

QuadriFlow: A Scalable and Robust Method for Quadrangulation

QuadriFlow is a scalable algorithm for generating quadrilateral surface meshes based on the Instant Field‐Aligned Meshes of Jakob et al. (ACM Trans. Graph. 34(6):189, 2015). We modify the original algorithm such that it efficiently produces meshes with many fewer singularities. Singularities in quadrilateral meshes cause problems for many applications, including parametrization and rendering with Catmull‐Clark subdivision surfaces. Singularities can rarely be entirely eliminated, but it is possible to keep their number small. Local optimization algorithms usually produce meshes with many singularities, whereas the best algorithms tend to require non‐local optimization, and therefore are slow. We propose an efficient method to minimize singularities by combining the Instant Meshes objective with a system of linear and quadratic constraints. These constraints are enforced by solving a global minimum‐cost network flow problem and local boolean satisfiability problems. We have verified the robustness and efficiency of our method on a subset of ShapeNet comprising 17,791 3D objects in the wild. Our evaluation shows that the quality of the quadrangulations generated by our method is as good as, if not better than, those from other methods, achieving about four times fewer singularities than Instant Meshes. Other algorithms that produce similarly few singularities are much slower; we take less than ten seconds to process each model. Our source code is publicly available.     

Linear Surface Reconstruction from Discrete Fundamental Forms on Triangle Meshes

We present a linear algorithm to reconstruct the vertex coordinates for a surface mesh given its edge lengths and dihedral angles, unique up to rotation and translation. A local integrability condition for the existence of an immersion of the mesh in 3D Euclidean space is provided, mirroring the fundamental theorem of surfaces in the continuous setting (i.e. Gauss's equation and the Mainardi–Codazzi equations) if we regard edge lengths as the discrete first fundamental form and dihedral angles as the discrete second fundamental form. The resulting sparse linear system to solve for the immersion is derived from the convex optimization of a quadratic energy based on a lift from the immersion in the 3D Euclidean space to the 6D rigid motion space. This discrete representation and linear reconstruction can benefit a wide range of geometry processing tasks such as surface deformation and shape analysis. A rotation‐invariant surface deformation through point and orientation constraints is demonstrated as well.