Scalable Remote Rendering with Depth and Motion‐flow Augmented Streaming

In this paper, we focus on efficient compression and streaming of frames rendered from a dynamic 3D model. Remote rendering and on‐the‐fly streaming become increasingly attractive for interactive applications. Data is kept confidential and only images are sent to the client. Even if the client's hardware resources are modest, the user can interact with state‐of‐the‐art rendering applications executed on the server. Our solution focuses on augmented video information, e.g., by depth, which is key to increase robustness with respect to data loss, image reconstruction, and is an important feature for stereo vision and other client‐side applications. Two major challenges arise in such a setup. First, the server workload has to be controlled to support many clients, second the data transfer needs to be efficient. Consequently, our contributions are twofold. First, we reduce the server‐based computations by making use of sparse sampling and temporal consistency to avoid expensive pixel evaluations. Second, our data‐transfer solution takes limited bandwidths into account, is robust to information loss, and compression and decompression are efficient enough to support real‐time interaction. Our key insight is to tailor our method explicitly for rendered 3D content and shift some computations on client GPUs, to better balance the server/client workload. Our framework is progressive, scalable, and allows us to stream augmented high‐resolution (e.g., HD‐ready) frames with small bandwidth on standard hardware.     

基于小波的网络多分辨率体绘制

为了能在网络环境下，快速地进行三维绘制，提出了一种网络环境下的基于小波的体数据多分辨率体绘制方法，该方法采取基于客户端的三维重建方案，首先利用三维小波的多分辨率分析方法，将体数据分解为不同分辨率下的离散逼近信号与高频细节信号；然后按先离散逼近信号，后高频细节信号的次序将数据传输到客户端；最后在客户端实现由粗及精的、渐进式的三维绘制。在这个过程中，一种3D的Mallat滤波器组被用来加速体数据的3D小波分解与重构，一种离散的简略化小波域体绘制方程被用来满足体绘制的实时性。实验结果表明，由于该方法仅需要12．5％或更低的数据量，即可以绘制出品质良好的图像或图像的概貌，所以非常适宜于需要频繁选择、交互的三维图像网络系统。     

Web-Based Remote Renderingwith IBRAC (Image-Based Rendering Acceleration and Compression)

Recent advances in Internet and computer graphics stimulate intensive use and development of 3D graphics on the World Wide Web. To increase efficiency of systems using 3D graphics on the web, the presented method utilizes previously rendered and transmitted images to accelerate the rendering and compression of new synthetic scene images. The algorithm employs ray casting and epipolar constraints to exploit spatial and temporal coherence between the current and previously rendered images. The reprojection of color and visibility data accelerates the computation of new images. The rendering method intrinsically computes a residual image, based on a user specified error tolerance that balances image quality against computation time and bandwidth. Encoding and decoding uses the same algorithm, so the transmitted residual image consists only of significant data without addresses or offsets. We measure rendering speed-ups of four to seven without visible degradation. Compression ratios per frame are a factor of two to ten better than MPEG2 in our test cases. There is no transmission of 3D scene data to delay the first image. The efficiency of the server and client generally increases with scene complexity or data size since the rendering time is predominantly a function of image size. This approach is attractive for remote rendering applications such as web-based scientific visualization where a client system may be a relatively low-performance machine and limited network bandwidth makes transmission of large 3D data impractical.     

基于Web的时变体数据的体绘制方法

传统Web体绘制方法主要集中在利用服务器端进行预处理和绘制任务,浏览器端仅用于呈现绘制结果,这样会造成服务器负载过高,同时,当绘制参数发生更改时,必须向服务器请求新的绘制结果,这样也易受网络延迟的影响。为了解决以上问题,实现在浏览器本地进行体绘制和交互,本文提出一种基于WebGL的体绘制方法,以时变体数据为例,在浏览器端实现光线投射体绘制算法。同时,为了提升绘制效率和减少内存占用,本文基于维度压缩方法,优化时变体数据的预处理过程。最后,本文设计了Web体绘制系统,引入暴风时变数据集以验证方法的有效性,结果表明,本文方法能够在浏览器本地对时变体数据进行体绘制,绘制时间在50ms以下,帧速率可达到50 FPS以上,同时支持实时交互,并且当绘制参数发生更改时,系统能够直接在浏览器端进行重新绘制。     

Image-Based Network Rendering of Large Meshes for Cloud Computing

Recent advances in sensing and software technologies enable us to obtain large-scale, yet fine 3D mesh models of cultural assets. However, such large models cannot be displayed interactively on consumer computers because of the performance limitation of the hardware. Cloud computing technology is a solution that can process a very large amount of information without adding to each client user’s processing cost. In this paper, we propose an interactive rendering system for large 3D mesh models, stored on a remote environment through a network of relatively small capacity machines, based on the cloud computing concept. Our system uses both model- and image-based rendering methods for efficient load balance between a server and clients. On the server, the 3D models are rendered by the model-based method using a hierarchical data structure with Level of Detail (LOD). On the client, an arbitrary view is constructed by using a novel image-based method, referred to as the Grid-Lumigraph, which blends colors from sampling images received from the server. The resulting rendering system can efficiently render any image in real time. We implemented the system and evaluated the rendering and data transferring performance.     

渲染器与Web服务器耦合实现远程体 渲染的交互优化

目的 为使用户随时随地获得需要较大计算和存储资源的交互真实感体渲染服务,设计并实现了一种面向Web的远程真实感体渲染方法。方法 计算量较大的实时体渲染任务由远端渲染服务器中的GPU加速完成并通过WebSocket将渲染图像发送至客户端;客户端浏览器只需负责接收显示渲染图像并监听发送用户交互事件。提出了一种输出系统耦合算法用以连接输出图像速率较大的渲染服务器和发送图像速率较慢的Web服务器。算法能根据Web服务器发送图像的情况动态调整每次图像输出的迭代计算次数,改变渲染服务器输出图像的时间间隔以达到与Web服务器发送速度相平衡,同时保持渲染服务器持续工作。结果 实验比较了在不同网络传输条件下,采用输出系统耦合算法与直接连接渲染器和Web服务器,渲染4个不同数据集所需的完成时间及帧率等性能评价指标。在局域网和广域网环境下,本文方法分别最多只需17 s和14 s即可完成整个渲染过程,而采用直接连接渲染器和Web服务器的方法则分别至少需要31 s和60 s才能完成整个渲染过程。实验结果表明采用输出系统耦合算法在不同网络条件下均可较大地缩短整个渲染过程所需时间,使用户在较短时间内获得高质量渲染图像。结论 本文渲染器与Web服务器耦合实现远程体渲染交互优化的方法可让用户使用与计算能力无关的桌面或移动设备通过网络透明使用高性能渲染系统;系统采用的输出系统耦合算法能够根据网络承载能力自适应调整渲染器输出速度,使用户在不同的网络环境中均可以较快的速度获得高质量渲染图像。     

Navigating large terrains using commodity mobile devices

We describe a hybrid client-server technique for remote adaptive streaming and rendering of large terrains in resource-limited mobile devices. The technique has been designed to achieve an interactive rendering performance on a mobile device connected to a low-bandwidth wireless network. The rendering workload is split between the client and the server. The terrain area close to the viewer is rendered in real-time by the client. The terrain located far from the viewer is portrayed as view-dependent impostors, rendered by the server on demand. A prototype has been built and an exhaustive set of experiments covering several platforms, wireless networks and a wide range of viewer velocities has been conducted. Results show that the approach is feasible, effective and robust.     

移动设备上基于三维模型的线条画绘制

提出一种综合考虑移动设备客户端绘制能力和传输速度的混合型远程绘制方法.该方法采用渐进网格得到不同分辨率的细节层次模型,并利用渐进传输机制逐步传输数据,以缓解单次网络负载.服务器端可以根据网络质量状况、移动客户端的处理能力和用户交互请求来选择适宜的三维细节层次模型,提取特征轮廓线并将生成的线条集合传输给客户端进行本地风格化绘制.将该方法应用于考古数字博物馆展示中取得了良好的效果.     

Render2MPEG: A Perception-based Framework Towards Integrating Rendering and Video Compression

Currently 3D animation rendering and video compression are completely independent processes even if rendered frames are streamed on‐the‐fly within a client‐server platform. In such scenario, which may involve time‐varying transmission bandwidths and different display characteristics at the client side, dynamic adjustment of the rendering quality to such requirements can lead to a better use of server resources. In this work, we present a framework where the renderer and MPEG codec are coupled through a straightforward interface that provides precise motion vectors from the rendering side to the codec and perceptual error thresholds for each pixel in the opposite direction. The perceptual error thresholds take into account bandwidth‐dependent quantization errors resulting from the lossy com‐pression as well as image content‐dependent luminance and spatial contrast masking. The availability of the discrete cosine transform (DCT) coefficients at the codec side enables to use advanced models of the human visual system (HVS) in the perceptual error threshold derivation without incurring any significant cost. Those error thresholds are then used to control the rendering quality and make it well aligned with the compressed stream quality. In our prototype system we use the lightcuts technique developed by Walter et al., which we enhance to handle dynamic image sequences, and an MPEG‐2 implementation. Our results clearly demonstrate many advantages of coupling the rendering with video compression in terms of faster rendering. Furthermore, temporally coherent rendering leads to a reduction of temporal artifacts.     

面向移动终端的分布并行化渲染

目的 随着移动互联技术和实时渲染技术的快速发展,面向移动终端的3维展示提供了远程交互式的模型渲染,但较高的渲染计算复杂度与较大的数据处理规模,影响了移动终端3维展示的渲染质量和实时性。针对以上不足,提出一种面向移动终端的分布并行化渲染方法。方法 该方法将渲染任务分布到服务端与终端,服务端采用层次细节模型控制场景复杂度,生成初次渲染图像;终端采用基于图像的渲染技术再次绘制图像,提高渲染质量,同时在渲染过程中利用CUDA(compute unified device architecture)并行计算加速渲染数据处理。结果 本方法有效提高了渲染速度,降低了数据传输量,并保证了图像质量,帧率和数据传输量优化了大约10.8%。结论 本文方法为面向移动终端的3维展示提供了很好的解决途径,在移动网络环境中,能够有效降低服务端负载压力,提高资源利用率并改善用户体验。     

Interactive visualization of medical volume models in mobile devices

Interactive visualization of volume models in standard mobile devices is a challenging present problem with increasing interest from new application fields like telemedicine. The complexity of present volume models in medical applications is continuously increasing, therefore increasing the gap between the available models and the rendering capabilities in low-end mobile clients. New and efficient rendering algorithms and interaction paradigms are required for these small platforms. In this paper, we propose a transfer function-aware compression and interaction scheme, for client-server architectures with visualization on standard mobile devices. The scheme is block-based, supporting adaptive ray-casting in the client. Our two-level ray-casting allows focusing on small details on targeted regions while keeping bounded memory requirements in the GPU of the client. Our approach includes a transfer function-aware compression scheme based on a local wavelet transformation, together with a bricking scheme that supports interactive inspection and levels of detail in the mobile device client. We also use a quantization technique that takes into account a perceptive metrics of the visual error. Our results show that we can have full interaction with high compression rates and with transmitted model sizes that can be of the order of a single photographic image.     

基于小波的图像序绘制算法研究

以三维医学ＣＴ数据为实验对象,描述基于小波的图像序列绘制的关键技术;物质人了数、体数数据的小波表示、明暗计算及体绘制积分的计算。修正现有文献中差异 分因子的定义,由此提出新的数据结构,阐述利用同一光线上相邻抽样点的相关性构造数据结构的技术,给出了简化的明暗计算方法,以重构三维数据场代替重构光亮度场,提出基于小 图像序绘制的改进算法。最后,通过实例检验了有关绘制算法的实用性能。     

Progressive volume rendering of large unstructured grids

We describe a new progressive technique that allows real-time rendering of extremely large tetrahedral meshes. Our approach uses a client-server architecture to incrementally stream portions of the mesh from a server to a client which refines the quality of the approximate rendering until it converges to a full quality rendering. The results of previous steps are re-used in each subsequent refinement, thus leading to an efficient rendering. Our novel approach keeps very little geometry on the client and works by refining a set of rendered images at each step. Our interactive representation of the dataset is efficient, light-weight, and high quality. We present a framework for the exploration of large datasets stored on a remote server with a thin client that is capable of rendering and managing full quality volume visualizations.     

基于视点的大规模点云数据实时预测调度策略

针对海量数据难以实时渲染的问题,提出基于视点的大规模点云数据实时预测调度策略——“5-5-5”动态调度策略。对整个三维点云进行“分区-子块化”处理,经过多分辨率压缩后保存到服务器端的数据库中。客户端根据视点的变化和菱形调度规则对子块数据进行内外存动态调度并实时绘制,同时向服务器发送传输新数据的请求。实验证明,该算法能有效解决漫游过程中的“抖动”现象,减少单次向服务器请求传输的数据量,实现漫游显示的连续性和平滑性。     

低熵图像序列无损压缩

大规模的云渲染技术带来了大量的三维图形渲染数据。为了减小集群渲染产生的图像序列数据的传输以及存储代价,针对渲染图像序列低熵的特点,基于字典编码技术提出了降低数据局部复杂性的无损数据压缩方案。该方案通过数据重排技术来大大提高数据的局部冗余度,从而提高数据无损压缩效率。为了进一步解决大规模图像序列的压缩耗时问题,提出了一种云计算平台上的分布式图像压缩处理方案,充分利用现有云计算中Map/Reduce计算模型实现了分布式编码方案。实验结果证明,对于渲染产生的大规模低熵图像序列,提出的方案能够有效提高编码率并减少编码时间。     

一种基于授权机制的分布式文件系统小文件访问优化策略

针对分布式文件系统应用于海量小文件访问模式时, 元数据请求过多导致系统性能下降的问题, 提出了客户端元数据缓存授权机制的解决方案. 客户端从服务器读取元数据时, 申请相应类型的授权, 服务器分析请求并决定是否授予此客户端所访问的元数据的授权. 若客户端成功获取授权, 则将其与本地缓存的元数据相关联, 作为缓冲有效性的凭证. 当再次访问本地缓存的元数据时, 若有相关授权, 则可以直接从本地获取元数据, 无需向服务器发送缓存数据的有效性验证RPC. 仿真实验表明, 文中的方法有效的降低了客户端发送元数据请求RPC的数量, 节省了宝贵的网络带宽资源, 降低了元数据服务器的负载.     

Metrics and generation specifications for comparing volume‐rendered images

The goal of this paper is to lay a foundation for objectively comparing volume‐rendered images, leading to objective evaluation of their accuracy and quality. The key elements of the foundation are: (1) a rigorous specification of all the input and parameters that need to be specified to define the conditions under which a volume‐rendered image is generated; and (2) the basis for a methodology for difference classification, including a suite of functions or metrics to quantify and classify the difference between two volume‐rendered images that will support an analysis of the relative importance of particular differences. The results of this method can be used to study the changes caused by modifying particular parameter values, to compare and quantify changes between images of similar data sets rendered in the same way, and to detect errors in the design, implementation or modification of a volume‐rendering system. If a benchmark image is available, for example one created by a high‐accuracy volume‐rendering system, the method can be used to evaluate the accuracy of a given image. The key contribution of this paper is the separation of the difference into noise, bias and structured difference components. Copyright © 1999 John Wiley & Sons, Ltd.     

三维地形模型流式传输方法

为满足基于网络的大规模地形漫游系统中有限的网络带宽与海量地形数据之间的矛盾,提出了一种三维地形流式渐进传输方法。该方法在服务器端对地形数据进行预处理,首先将地形数据进行分块,再对分块后的数据进行小波变换,最后用多级树集合分裂(SPIHT)算法将变换后的数据组织成多条渐进码流。客户端在运行时根据视点的位置,漫游的速度和方向确定周围各地块的传输规则,使各地块的传输量进行自适应调整,同时采用二级缓存机制缓解网络拥塞时数据供应压力。实验表明,该方法能很好地支持客户端基于GPU的多分辨率地形构造并在各种网络带宽情况下保持较高的绘制帧率,保证网络交互绘制的实时性。     

基于小波的三维图像频域显示方法研究

提出了一种基于小波的频域体绘制算法(FWVR)进行三维图像重构。算法首先完成三维体数据的傅里叶变换和切平面提取,然后在频域中完成小波的多分辨率分解,并在不同分辨级上进行小波系数重建,最后做傅里叶逆变换,以得到观察平面上希望得到的体绘制图像。算法是基于CT成像和小波变换的原理,时间复杂度由视平面上的二维傅里叶逆变换决定,即O(N2logN)。应用此方法对128的医学体数据进行实验,结果证明该方法有效,采用伪彩色后的绘制效果3与光线投射法绘制效果基本相同,而速度大大提高。     

一种基于蒙特卡罗方法的体绘制算法*

为解决医学图像三维可视化中大规模体数据显示速度与成像质量问题,引入一种基于蒙特卡罗方法的新颖体绘制算法。本算法根据给出的概率密度函数从随机样本点或其子集中选取一个点阵来进行绘制,主要适用于对大规模体数据集的有效可视化。另外,使用渐进细化的方法在图像质量和交互性上进行权衡折中,以适应实时要求。实验结果表明,在最终图像分辨率固定的情况下,本算法的时间复杂度和空间复杂度都比之前的算法要好,成像质量也满足实时绘制要求,可以在实际应用中使用。