Rendering localized spatial audio in a virtual auditory space

High-quality virtual audio scene rendering is required for emerging virtual and augmented reality applications, perceptual user interfaces, and sonification of data. We describe algorithms for creation of virtual auditory spaces by rendering cues that arise from anatomical scattering, environmental scattering, and dynamical effects. We use a novel way of personalizing the head related transfer functions (HRTFs) from a database, based on anatomical measurements. Details of algorithms for HRTF interpolation, room impulse response creation, HRTF selection from a database, and audio scene presentation are presented. Our system runs in real time on an office PC without specialized DSP hardware.     

声学仿真中的人工神经网络方法

虽然许多研究人员已认识到三维真实感声音在未来人机交互中的重要地位，但是三维真实感声音在计算机领域的真正实现仍有不少障碍有待克服．基于对声学及心理声学最新研究成果的调查和分析，本文设计并实现了一个基于神经网络方法的ＨＲＴＦ（ｈｅａｄ－ｒｅｌａｔｅｄｔｒａｎｓｆｅｒｆｕｎｃｔｉｏｎ）模型，用于三维真实感声音的生成．模型中的数据可通过网络学习进行重新设置，以满足多种场合的需要．并且，通过神经网络的非线性拟合能力，可以获取空间任意位置的ＨＲＴＦ数据．初步的实验表明了该方法的有效性和正确性．     

Three-dimensional shape rendering from multiple images

A paradigm for automatic three-dimensional shape and geometry rendering from multiple images is introduced in this paper. In particular, non-photorealistic rendering (NPR) techniques in the style of pen-and-ink illustrations are addressed, while the underlying presented ideas can be used in other modalities, such as halftoning, as well. Existing NPR approaches can be categorized in two groups depending on the type of input they use: image based and object based. Using multiple images as input to the NPR scheme, we propose a novel hybrid model that simultaneously uses information from the image and object domains. The benefit not only comes from combining the features of each approach, it also minimizes the need for manual or user assisted tasks in extracting scene features and geometry, as employed in virtually all state-of-the-art NPR approaches. As particular examples we use input images from binocular stereo and multiple-light photometric stereo systems. From the image domain we extract the tonal information to be mimicked by the NPR synthesis algorithm, and from the object domain we extract the geometry, mainly principal directions, obtained from the image set without explicitly using 3D models, to convey shape to the drawings. We describe a particular implementation of such an hybrid system and present a number of automatically generated pen-and-ink style drawings. This work then shows how to use and extend well-developed techniques in computer vision to address fundamental problems in shape representation and rendering.     

Articulated Billboards for Video‐based Rendering

We present a novel representation and rendering method for free‐viewpoint video of human characters based on multiple input video streams. The basic idea is to approximate the articulated 3D shape of the human body using a subdivision into textured billboards along the skeleton structure. Billboards are clustered to fans such that each skeleton bone contains one billboard per source camera. We call this representation articulated billboards. In the paper we describe a semi‐automatic, data‐driven algorithm to construct and render this representation, which robustly handles even challenging acquisition scenarios characterized by sparse camera positioning, inaccurate camera calibration, low video resolution, or occlusions in the scene. First, for each input view, a 2D pose estimation based on image silhouettes, motion capture data, and temporal video coherence is used to create a segmentation mask for each body part. Then, from the 2D poses and the segmentation, the actual articulated billboard model is constructed by a 3D joint optimization and compensation for camera calibration errors. The rendering method includes a novel way of blending the textural contributions of each billboard and features an adaptive seam correction to eliminate visible discontinuities between adjacent billboards textures. Our articulated billboards do not only minimize ghosting artifacts known from conventional billboard rendering, but also alleviate restrictions to the setup and sensitivities to errors of more complex 3D representations and multiview reconstruction techniques. Our results demonstrate the flexibility and the robustness of our approach with high quality free‐viewpoint video generated from broadcast footage of challenging, uncontrolled environments.     

Online Personalised Non‐photorealistic Rendering Technique for 3D Geometry from Incremental Sketching

This paper presents an online personalised non‐photorealistic rendering (NPR) technique for 3D models generated from interactively sketched input. This technique has been integrated into a sketch‐based modelling system. It lets users interact with computers by drawing naturally, without specifying the number, order, or direction of strokes. After sketches are interpreted as 3D objects, they can be rendered with personalised drawing styles so that the reconstructed 3D model can be presented in a sketchy style similar in appearance to what have been drawn for the 3D model. This technique captures the user's drawing style without using template or prior knowledge of the sketching style. The personalised rendering style can be applied to both visible and initially invisible geometry. The rendering strokes are intelligently selected from the input sketches and mapped to edges of the 3D object. In addition, non‐geometric information such as surface textures can be added to the recognised object in different sketching modes. This will integrate sketch‐based incremental 3D modelling and NPR into conceptual design.     

Semi‐Supervised Learning in Reconstructed Manifold Space for 3D Caricature Generation

Recently, automatic 3D caricature generation has attracted much attention from both the research community and the game industry. Machine learning has been proven effective in the automatic generation of caricatures. However, the lack of 3D caricature samples makes it challenging to train a good model. This paper addresses this problem by two steps. First, the training set is enlarged by reconstructing 3D caricatures. We reconstruct 3D caricatures based on some 2D caricature samples with a Principal Component Analysis (PCA)‐based method. Secondly, between the 2D real faces and the enlarged 3D caricatures, a regressive model is learnt by the semi‐supervised manifold regularization (MR) method. We then predict 3D caricatures for 2D real faces with the learnt model. The experiments show that our novel approach synthesizes the 3D caricature more effectively than traditional methods. Moreover, our system has been applied successfully in a massive multi‐user educational game to provide human‐like avatars.     

ClipFlip : Multi‐view Clipart Design

We present an assistive system for clipart design by providing visual scaffolds from the unseen viewpoints. Inspired by the artists' creation process, our system constructs the visual scaffold by first synthesizing the reference 3D shape of the input clipart and rendering it from the desired viewpoint. The critical challenge of constructing this visual scaffold is to generate a reference 3D shape that matches the user's expectations in terms of object sizing and positioning while preserving the geometric style of the input clipart. To address this challenge, we propose a user‐assisted curve extrusion method to obtain the reference 3D shape. We render the synthesized reference 3D shape with a consistent style into the visual scaffold. By following the generated visual scaffold, the users can efficiently design clipart with their desired viewpoints. The user study conducted by an intuitive user interface and our generated visual scaffold suggests that our system is especially useful for estimating the ratio and scale between object parts and can save on average 57% of drawing time.     

基于手绘输入理解的智能3D几何建模系统

近年来,图形学领域已经开发了大量的3D建模和可视化工具,但是这些工具的界面往往过于复杂,制约了设计人员的创造性工作,也进一步限制了这些工具为普遍用户所使用。事实上,理想的3D建模工具应该提供给用户以尽可能自然、简单的操作,例如基于电子笔的草图输入,而且随着移动设备的发展,也使之成为图形学中更有前景的研究方向。文章论述了当前支持手绘的建模技术的研究现状,并提出了一个更加智能化的多模块建模体系,突出了交互技术、智能理解和高质量的可视化输出技术的结合;而且针对手绘输入的不明确性,着重分析了图形数据的基于特性的语义描述,提出了智能的逻辑推理模型,并进一步描述了下一代融合自然输入、逻辑推理与虚拟显示于一体的智能图形建模系统的实现。预测了人工智能技术与CA D技术的结合将是未来图形学领域发展的主要方向之一。     

Silhouette‐Aware Warping for Image‐Based Rendering

Image‐based rendering (IBR) techniques allow capture and display of 3D environments using photographs. Modern IBR pipelines reconstruct proxy geometry using multi‐view stereo, reproject the photographs onto the proxy and blend them to create novel views. The success of these methods depends on accurate 3D proxies, which are difficult to obtain for complex objects such as trees and cars. Large number of input images do not improve reconstruction proportionally; surface extraction is challenging even from dense range scans for scenes containing such objects. Our approach does not depend on dense accurate geometric reconstruction; instead we compensate for sparse 3D information by variational image warping. In particular, we formulate silhouette‐aware warps that preserve salient depth discontinuities. This improves the rendering of difficult foreground objects, even when deviating from view interpolation. We use a semi‐automatic step to identify depth discontinuities and extract a sparse set of depth constraints used to guide the warp. Our framework is lightweight and results in good quality IBR for previously challenging environments.     

Auditory scene reproduction for tele-operated robot systems

ABSTRACT

In a tele-operated robot environment, reproducing auditory scenes and conveying 3D spatial information of sound sources are inevitable in order to make operators feel more realistic tele-presence. In this paper, we propose a tele-presence robot system that enables reproduction and manipulation of auditory scenes. This tele-presence system is carried out on the basis of 3D information about where targeted humans are speaking, and matching with the operator's head orientation. We employed multiple microphone arrays and human tracking technologies to localize and separate voices around a robot. In the operator site, separated sound sources are rendered using head-related transfer functions (HRTF) according to the sound sources' spatial positions and the operator's head orientation that is being tracked real time. Two-party and three-party interaction experiments indicated that the proposed system has significantly higher accuracy when perceiving direction of sounds and gains higher subjective scores in sense of presence and listenability, compared to a baseline system which uses stereo binaural sounds obtained by two microphones mounted on the humanoid robot's ears.     

Peek‐in‐the‐Pic: Flying Through Architectural Scenes From a Single Image*

Many casually taken ‘tourist’ photographs comprise of architectural objects like houses, buildings, etc. Reconstructing such 3D scenes captured in a single photograph is a very challenging problem. We propose a novel approach to reconstruct such architectural scenes with minimal and simple user interaction, with the goal of providing 3D navigational capability to an image rather than acquiring accurate geometric detail. Our system, Peek‐in‐the‐Pic, is based on a sketch‐based geometry reconstruction paradigm. Given an image, the user simply traces out objects from it. Our system regards these as perspective line drawings, automatically completes them and reconstructs geometry from them. We make basic assumptions about the structure of traced objects and provide simple gestures for placing additional constraints. We also provide a simple sketching tool to progressively complete parts of the reconstructed buildings that are not visible in the image and cannot be automatically completed. Finally, we fill holes created in the original image when reconstructed buildings are removed from it, by automatic texture synthesis. Users can spend more time using interactive texture synthesis for further refining the image. Thus, instead of looking at flat images, a user can fly through them after some simple processing. Minimal manual work, ease of use and interactivity are the salient features of our approach.     

From on-line sketching to 2D and 3D geometry: a system based on fuzzy knowledge

The paper describes the development of a fuzzy knowledge-based prototype system for conceptual design. This real time system is designed to infer user's sketching intentions, to segment sketched input and generate corresponding geometric primitives: straight lines, circles; arcs, ellipses, elliptical arcs, and B-spline curves. Topology information (connectivity, unitary constraints and pairwise constraints) is received dynamically from 2D sketched input and primitives. From the 2D topology information, a more accurate 2D geometry can be built up by applying a 2D geometric constraint solver. Subsequently, 3D geometry can be received feature by feature incrementally. Each feature can be recognised by inference knowledge in terms of matching its 2D primitive configurations and connection relationships. The system accepts not only sketched input, working as an automatic design tool, but also accepts user interactive input of both 2D primitives and special positional 3D primitives. This makes it easy and friendly to use. The system has been tested with a number of sketched inputs of 2D and 3D geometry.     

Learning Probabilistic Transfer Functions: A Comparative Study of Classifiers

Complex volume rendering tasks require high‐dimensional transfer functions, which are notoriously difficult to design. One solution to this is to learn transfer functions from scribbles that the user places in the volumetric domain in an intuitive and natural manner. In this paper, we explicitly model and visualize the uncertainty in the resulting classification. To this end, we extend a previous intelligent system approach to volume rendering, and we systematically compare five supervised classification techniques – Gaussian Naive Bayes, k Nearest Neighbor, Support Vector Machines, Neural Networks, and Random Forests – with respect to probabilistic classification, support for multiple materials, interactive performance, robustness to unreliable input, and easy parameter tuning, which we identify as key requirements for the successful use in this application. Based on theoretical considerations, as well as quantitative and visual results on volume datasets from different sources and modalities, we conclude that, while no single classifier can be expected to outperform all others under all circumstances, random forests are a useful off‐the‐shelf technique that provides fast, easy, robust and accurate results in many scenarios.     

Single‐image Tomography: 3D Volumes from 2D Cranial X‐Rays

As many different 3D volumes could produce the same 2D x‐ray image, inverting this process is challenging. We show that recent deep learning‐based convolutional neural networks can solve this task. As the main challenge in learning is the sheer amount of data created when extending the 2D image into a 3D volume, we suggest firstly to learn a coarse, fixed‐resolution volume which is then fused in a second step with the input x‐ray into a high‐resolution volume. To train and validate our approach we introduce a new dataset that comprises of close to half a million computer‐simulated 2D x‐ray images of 3D volumes scanned from 175 mammalian species. Future applications of our approach include stereoscopic rendering of legacy x‐ray images, re‐rendering of x‐rays including changes of illumination, view pose or geometry. Our evaluation includes comparison to previous tomography work, previous learning methods using our data, a user study and application to a set of real x‐rays.     

Efficient opacity specification based on feature visibilities in direct volume rendering

Due to 3D occlusion, the specification of proper opacities in direct volume rendering is a time‐consuming and unintuitive process. The visibility histograms introduced by Correa and Ma reflect the effect of occlusion by measuring the influence of each sample in the histogram to the rendered image. However, the visibility is defined on individual samples, while volume exploration focuses on conveying the spatial relationships between features. Moreover, the high computational cost and large memory requirement limits its application in multi‐dimensional transfer function design. In this paper, we extend visibility histograms to feature visibility, which measures the contribution of each feature in the rendered image. Compared to visibility histograms, it has two distinctive advantages for opacity specification. First, the user can directly specify the visibilities for features and the opacities are automatically generated using an optimization algorithm. Second, its calculation requires only one rendering pass with no additional memory requirement. This feature visibility based opacity specification is fast and compatible with all types of transfer function design. Furthermore, we introduce a two‐step volume exploration scheme, in which an automatic optimization is first performed to provide a clear illustration of the spatial relationship and then the user adjusts the visibilities directly to achieve the desired feature enhancement. The effectiveness of this scheme is demonstrated by experimental results on several volumetric datasets.     

基于中国人工头模的HRTF数据库设计与实现

头相关传输函数(Head Related Transfer Function,HRTF)描述了在自由场情况下,点声源到人耳鼓膜处的传输过程,其中包含有重要的声源定位信息。本文搭建HRTF测量与实验环境,设计和实现一个HRTF数据库,数据库包含中国人平均头模BHead210 481个空间方位的头相关脉冲响应(Head Related Impulse Response)数据。进行主观定位判听实验,比较BHead210人工头测量的HRTF数据和KEMAR人工头HRTF数据在中国受试者上的判听效果。     

3D人体面部表情动画系统

本文讨论动画系统的一个子集的发展.中间目标是产生一个系统,它基于人体面部的视频输入,能实时产生人面的3D模型动画.首要的主题是模型操作、设计和配合、声频视频操作和同步、3D模型演示和表演、图像处理和特征提取.用连续的红绿蓝图像,在一个时间抓一帧画面,图像处理被计时.实验表明,用全面搜索定位面部特征大约需要70ms,用局部搜索定位特征大约5ms.考虑到模型显示,模型显示用OpenInventor成功完成.     

A modified radiosity algorithm for integrated visual and auditory rendering

A concept on integration of 3D visual and auditory rendering is proposed in this paper. By the auditory rendering we mean the computer acoustical simulation with direct sound outputs that is based on the physical model simulation. The so-called visual rendering means the computer generation of 3D images of real world and even virtual world, but without sounds. By integration of 3D visual and auditory rendering we mean while computer generates a 3D image of physical model of real world, it generates the sound based on the physical model displayed if there is a reverberation-free sound source. Based on some kinds of similarity between the propagation nature of light and sound, a modified radiosity algorithm has been proposed for integrating implementation. The algorithm is suitable for both visual and room auditory rendering. It means for a given enclosure, once the form factors are calculated, they can be applied for calculating both the light radiosities and sound radiosities simultaneously. The features of the algorithm are indicated. The implementation techniques are given in detail. The integration of visual and auditory rendering is a big step toward real world simulation, furthermore it will extend the research direction of visualization in scientific computing (ViSC) to visualization and audiobilization in scientific computing (VAiSC). The combination of VAiSC with virtual reality technique will improve human-computer interface greatly.     

笔式三维草图绘制中的轮廓线技术研究*

详细介绍了三维模型轮廓线技术及其检测算法,并深入分析了轮廓线技术在三维草图绘制中的作用;然后设计并实现了一个三维草图绘制系统。该系统提供了用笔绘制三维模型的交互环境,通过轮廓线将三维模型与二维交互手势联系起来,使用户可以自由灵活地操纵和控制三维模型,获得自然、高效的交互体验。     

基于模型的人体动画方法

文章从点点对应,线线对应入手,首先给出了根据已知二维影像特征求得与其对应的三维模型特征参数的方法,然后给出了已知人体二维影像特征求解三维人体模型特征参数方法的详细描述.基于这一思想,实现了一个动画系统Video & Animation Studio.文章最后给出了一个由该系统实现的人体行走的例子.