Scalable Remote Rendering with Depth and Motion‐flow Augmented Streaming

In this paper, we focus on efficient compression and streaming of frames rendered from a dynamic 3D model. Remote rendering and on‐the‐fly streaming become increasingly attractive for interactive applications. Data is kept confidential and only images are sent to the client. Even if the client's hardware resources are modest, the user can interact with state‐of‐the‐art rendering applications executed on the server. Our solution focuses on augmented video information, e.g., by depth, which is key to increase robustness with respect to data loss, image reconstruction, and is an important feature for stereo vision and other client‐side applications. Two major challenges arise in such a setup. First, the server workload has to be controlled to support many clients, second the data transfer needs to be efficient. Consequently, our contributions are twofold. First, we reduce the server‐based computations by making use of sparse sampling and temporal consistency to avoid expensive pixel evaluations. Second, our data‐transfer solution takes limited bandwidths into account, is robust to information loss, and compression and decompression are efficient enough to support real‐time interaction. Our key insight is to tailor our method explicitly for rendered 3D content and shift some computations on client GPUs, to better balance the server/client workload. Our framework is progressive, scalable, and allows us to stream augmented high‐resolution (e.g., HD‐ready) frames with small bandwidth on standard hardware.     

A Collaborative Digital Pathology System for Multi‐Touch Mobile and Desktop Computing Platforms

Collaborative slide image viewing systems are becoming increasingly important in pathology applications such as telepathology and E‐learning. Despite rapid advances in computing and imaging technology, current digital pathology systems have limited performance with respect to remote viewing of whole slide images on desktop or mobile computing devices. In this paper we present a novel digital pathology client–server system that supports collaborative viewing of multi‐plane whole slide images over standard networks using multi‐touch‐enabled clients. Our system is built upon a standard HTTP web server and a MySQL database to allow multiple clients to exchange image and metadata concurrently. We introduce a domain‐specific image‐stack compression method that leverages real‐time hardware decoding on mobile devices. It adaptively encodes image stacks in a decorrelated colour space to achieve extremely low bitrates (0.8 bpp) with very low loss of image quality. We evaluate the image quality of our compression method and the performance of our system for diagnosis with an in‐depth user study.     

Web-Based Remote Renderingwith IBRAC (Image-Based Rendering Acceleration and Compression)

Recent advances in Internet and computer graphics stimulate intensive use and development of 3D graphics on the World Wide Web. To increase efficiency of systems using 3D graphics on the web, the presented method utilizes previously rendered and transmitted images to accelerate the rendering and compression of new synthetic scene images. The algorithm employs ray casting and epipolar constraints to exploit spatial and temporal coherence between the current and previously rendered images. The reprojection of color and visibility data accelerates the computation of new images. The rendering method intrinsically computes a residual image, based on a user specified error tolerance that balances image quality against computation time and bandwidth. Encoding and decoding uses the same algorithm, so the transmitted residual image consists only of significant data without addresses or offsets. We measure rendering speed-ups of four to seven without visible degradation. Compression ratios per frame are a factor of two to ten better than MPEG2 in our test cases. There is no transmission of 3D scene data to delay the first image. The efficiency of the server and client generally increases with scene complexity or data size since the rendering time is predominantly a function of image size. This approach is attractive for remote rendering applications such as web-based scientific visualization where a client system may be a relatively low-performance machine and limited network bandwidth makes transmission of large 3D data impractical.     

基于ARM11的H.264硬编解码视频传输系统设计

为了解决视频信息冗余、视频传输系统成本较高,视频编解码效率低等缺陷,研究了一种基于H.264硬编解码器的视频传输系统;该系统采用以ARM11为核心的,包含多媒体硬编解码器MFC的S3C6410作为处理器,使用CMOS摄像头采集实时视频数据;在嵌入式Linux操作系统上构建基于live555的RTSP服务器,监听客户端请求;重点阐述了多媒体硬编解码器MFC对视频数据进行的H.264硬件压缩编码,以及数据包的RTP封装,然后经由live555流媒体服务器转发至PC机;最后,在PC上对接收到的数据流进行解码播放,经实验测试证明,系统设计稳定可靠,具有可扩展性强、性能高、成本和功耗低等优点,图像质量和时延满足实际应用需求。     

Proxy‐guided Image‐based Rendering for Mobile Devices

VR headsets and hand‐held devices are not powerful enough to render complex scenes in real‐time. A server can take on the rendering task, but network latency prohibits a good user experience. We present a new image‐based rendering (IBR) architecture for masking the latency. It runs in real‐time even on very weak mobile devices, supports modern game engine graphics, and maintains high visual quality even for large view displacements. We propose a novel server‐side dual‐view representation that leverages an optimally‐placed extra view and depth peeling to provide the client with coverage for filling disocclusion holes. This representation is directly rendered in a novel wide‐angle projection with favorable directional parameterization. A new client‐side IBR algorithm uses a pre‐transmitted level‐of‐detail proxy with an encaging simplification and depth‐carving to maintain highly complex geometric detail. We demonstrate our approach with typical VR / mobile gaming applications running on mobile hardware. Our technique compares favorably to competing approaches according to perceptual and numerical comparisons.     

基于视觉的模型在图象压缩中应用的研究

针对在图象处理中对压缩和解压缩方法的评价缺乏一个与人类视觉相关性很好的模型这一问题,提出并使用了基于人类视觉的模型,并使用在lena和 fruit这两幅图象处理中的经典图片上,取得了和人类观测结果吻合的结论。     

A pretreatment to improve the quality metrics performance for encoding H264/AVC

The study of the human visual system is very interesting to quantify the quality of an image or to predict perceived information. The contrast sensitivity function is one of the main ways to incorporate the human visual system properties in an imaging system. It characterizes its sensitivity to spatial frequencies. In this paper, we are interested in establishing a pretreatment for existing metrics with full reference (“peak signal‐to‐noise ratio”, “digital video quality”) for the H.264/MPEG‐4 (Motion Picture Expert Group) advanced video coding standard. We realize in our algorithm the FFT transformation to apply the contrast sensitivity function. Our method is applicable to any size of image and video sequence by increasing its size at powers of two. This increase is achieved by adding “mirror image.” We evaluate the performance of the proposed pretreatment by using subjective “LIVE” video databases. The performance metrics, that is, Pearson (PLCC), Spearman correlation coefficients (SROCC) and root mean square prediction error (RMSE) indicate that the proposed method gives a good performance in H264 codec distortions.     

A reduced reference video quality assessment of H.264 and mpeg2 codec based on sharpness metric

In this paper, we propose a new perceptually significant video quality metric for the H.264/Motion Picture Expert Group (MPEG)‐4 Advanced Video Coding (AVC) and MPEG2 standard. Our method operates in the spatial domain by using the Sobel filter. The proposed approach does not require a high computational complexity and can be suitable for real‐time evaluation. We evaluate the performance of the proposed method by using three Common Intermediate Format sequences at different compression rates. The comparison of the obtained results is made with some video quality models using “LIVE”, “IVP” and “IRCCyN/IVC 1080i” databases. The performance metrics, i.e. Pearson and Spearman correlation coefficients, indicate that the proposed method gives a good performance in H264 and MPEG2 codec distortions with the three databases comparing with other models.     

4D video textures for interactive character appearance

4D Video Textures (4DVT) introduce a novel representation for rendering video‐realistic interactive character animation from a database of 4D actor performance captured in a multiple camera studio. 4D performance capture reconstructs dynamic shape and appearance over time but is limited to free‐viewpoint video replay of the same motion. Interactive animation from 4D performance capture has so far been limited to surface shape only. 4DVT is the final piece in the puzzle enabling video‐realistic interactive animation through two contributions: a layered view‐dependent texture map representation which supports efficient storage, transmission and rendering from multiple view video capture; and a rendering approach that combines multiple 4DVT sequences in a parametric motion space, maintaining video quality rendering of dynamic surface appearance whilst allowing high‐level interactive control of character motion and viewpoint. 4DVT is demonstrated for multiple characters and evaluated both quantitatively and through a user‐study which confirms that the visual quality of captured video is maintained. The 4DVT representation achieves >90% reduction in size and halves the rendering cost.     

Scene-effect detection and insertion MPEG encoding scheme for video browsing and error concealment

For an MPEG coding scheme, the encoder can do more than video compression. In this paper, a novel MPEG codec embedded with scene-effect information detection and insertion is proposed to provide more functionality at the decoder end. Based on the macroblock (MB) type of information that is generated simultaneously in the encoding process, a single-pass automatic scene-effect insertion MPEG coding scheme can be achieved. Using the USER/spl I.bar/DATA of picture header, the video output bitstreams by our method still conform to the conventional MPEG decoding system. The proposed method provides a solution toward upgrading the existing MPEG codec with low complexity to accomplish at least two major advantages. Precise and effective video browsing resulting from the scene-effect extraction can significantly reduce the user's time to look up what they are interested in. For video transmission, the bitstreams containing scene-effect information can obtain better error concealment performance when scene changes are involved. Compared with the gain it achieves, the payout of our algorithm is very worthy with comparatively small efforts.     

分布式多媒体系统中的媒体同步

本文介绍分布式多媒体系统中的媒体同步问题，着重介绍了基于MPEG－I的分布式多媒体系统中，视频和音频的同步以及媒体服务器和客户之间的流同步。     

Compression Domain Volume Rendering for Distributed Environments

This paper describes a method for volume data compression and rendering which bases on wavelet splats. The underlying concept is especially designed for distributed and networked applications, where we assume a remote server to maintain large scale volume data sets, being inspected, browsed through and rendered interactively by a local client. Therefore, we encode the server’s volume data using a newly designed wavelet based volume compression method. A local client can render the volumes immediately from the compression domain by using wavelet footprints, a method proposed earlier. In addition, our setup features full progression, where the rendered image is refined progressively as data comes in. Furthermore, framerate constraints are considered by controlling the quality of the image both locally and globally depending on the current network bandwidth or computational capabilities of the client. As a very important aspect of our setup, the client does not need to provide storage for the volume data and can be implemented in terms of a network application. The underlying framework enables to exploit all advantageous properties of the wavelet transform and forms a basis for both sophisticated lossy compression and rendering. Although coming along with simple illumination and constant exponential decay, the rendering method is especially suited for fast interactive inspection of large data sets and can be supported easily by graphics hardware.     

Sjtums流媒体点播协议和点播系统设计

传统的MMS和RTSP点播协议只能点播其自有的wmv和rm文件格式,但其不能提供有效的针对屏幕的视频编解码器.该文设计了一种高效的CSMX屏幕编解码器,同时压缩视频、音频,生成了一种自有格式的多媒体课件.为支持点播,设计并实现了一种新型流媒体点播协议sjtums协议以及整个流媒体点播系统.该协议交互原语简洁,每个客户端与点播服务器之间只创建一个连接,交互原语采用同步方式收发,流媒体数据采用异步方式收发,点播服务器采用多线程方式设计.客户端解决了三路流媒体之间的同步,点播文件的缓存以及视频的优化显示等难题.经过系统仿真和实际测试使用,该系统能够支持大规模客户端,点播效果稳定流畅.该点播协议的实现对流媒体协议的研究有着重要的理论价值.     

面向移动终端的分布并行化渲染

目的 随着移动互联技术和实时渲染技术的快速发展,面向移动终端的3维展示提供了远程交互式的模型渲染,但较高的渲染计算复杂度与较大的数据处理规模,影响了移动终端3维展示的渲染质量和实时性。针对以上不足,提出一种面向移动终端的分布并行化渲染方法。方法 该方法将渲染任务分布到服务端与终端,服务端采用层次细节模型控制场景复杂度,生成初次渲染图像;终端采用基于图像的渲染技术再次绘制图像,提高渲染质量,同时在渲染过程中利用CUDA(compute unified device architecture)并行计算加速渲染数据处理。结果 本方法有效提高了渲染速度,降低了数据传输量,并保证了图像质量,帧率和数据传输量优化了大约10.8%。结论 本文方法为面向移动终端的3维展示提供了很好的解决途径,在移动网络环境中,能够有效降低服务端负载压力,提高资源利用率并改善用户体验。     

基于H.264的嵌入式视频编码器的设计与应用

设计了一种适用于工业现场的嵌入式视频编码器，视频压缩按照H.264标准，在Intel 推出的PXA255 Xscale架构的处理机和源码开放的Linux嵌入式操作系统的基础上，实现了视频的实时动态采集、压缩和网络传输，介绍了系统的研制过程，给出了系统架构和关键技术。通过构建网络测试平台，将此编码器接入以太网，服务器端将接收到的压缩码流进行实时解压、回放。此编码器可用于远程视频监控系统中前端视频的实时采集、压缩和跨IP网传输。     

Video compression using MPEG 7-MBBMC: techniques for communication

Communication is termed as exchanging the information (audio, video, text and image) from one end (transmitter) to another end (receiver). When video data are compressed and transmitted to another side, compression reduces the bandwidth size and memory required to transmit the video. Some traditional techniques are used in video transmission but it includes drawbacks, such as more compression time and low quality due to compression. To overcome these drawbacks the MPEG7-MBBMC (Modified Block Based Motion Compensated) technique is developed. Here the input video signals are collected from the dataset and the signals are splitted into three bands. Discrete Wavelet Transform (DWT) is applied for each band and quantization process occurs. The DWT and quantization process are applied in the MPEG7 compression, which offers high compression factors. Next, encoder is applied to convert the packets into small packets by using modified block based motion compensated (MBBMC) technique. The Motion compensation establishes a correspondence between elements of nearby images in the video sequence. The Forward Error Correction (FEC) is used to reduce the distortion in the encoder video packet. Then the Channel Pattern Integration (CPI) is applied to find the best available channel. The encoded video packets are transmitted by the best available channel. In receiver side the error correction code is applied to decode the video packets and reconstructs the decoded packet by decompression. It improves the quality of the video and in future it will help for much development in the field of multimedia.

     

State‐of‐the‐Art Report in Web‐based Visualization

In this report, we review the current state of the art of web‐based visualization applications. Recently, an increasing number of web‐based visualization applications have emerged. This is due to the fact that new technologies offered by modern browsers greatly increased the capabilities for visualizations on the web. We first review these technical aspects that are enabling this development. This includes not only improvements for local rendering like WebGL and HTML5, but also infrastructures like grid or cloud computing platforms. Another important factor is the transfer of data between the server and the client. Therefore, we also discuss advances in this field, for example methods to reduce bandwidth requirements like compression and other optimizations such as progressive rendering and streaming. After establishing these technical foundations, we review existing web‐based visualization applications and prototypes from various application domains. Furthermore, we propose a classification of these web‐based applications based on the technologies and algorithms they employ. Finally, we also discuss promising application areas that would benefit from web‐based visualization and assess their feasibility based on the existing approaches.     

A new DCT audio watermarking scheme based on preliminary MP3 study

In this paper, a new audio watermarking scheme operating in the frequency domain and based on neural network architecture is described. The watermark is hidden into the middle frequency band after performing a Discrete Cosine transform (DCT). Embedding and extraction of the watermark are based on the use of a back-propagation neural network (BPNN) architecture. In addition, the selection of frequencies and the block hiding the watermark are based on a preliminary study of the effect of MP3 compression at several rates on the signal. Experimental results show that the proposed technique presents good robustness and perceptual quality results. We also investigate the application of the proposed technique in video watermarking. Traditional techniques have used audio channel as supplementary embedding space and adopt state-of-the art techniques that resist to MP3 compression attack. In these techniques, the MPEG compression attack is only evaluated on the video part and the audio part is kept unaffected. In this paper, we adapt the preliminary MP3 study to video watermarking technique but with a preliminary study of the MPEG compression applied to the audio channel. Here again, we notice that the application of the preliminary MPEG study to the audio channel improves the robustness of the video watermarking scheme though keeping high-quality watermarked video sequences.     

数据驱动的并行视频编码器的设计和实现(英文)

通用视频编码系统一般基于标准视频压缩算法, 比如 MPEG4 和 H.264。但这些标准算法包括如离散余弦变换和运动估计等高复杂度的计算,所以为了实现实时通信,编码系统通常采用以硬件为基础的实现方法。然而以硬件为基础的视频编码系统具有实施周期长、成本高、灵活性差等不足。文章介绍了一种基于低计算复杂度自适应块截短编码方案的数据驱动并行视频编码系统,实验结果验证了编码系统的有效性。     

基于大屏幕的交互式协同系统的设计与实现

介绍一套基于大屏幕的交互式协同系统,该系统能够在本地的大屏幕上显示多个远程高性能绘制系统的输出结果,为一组本地用户提供丰富的信息资源.系统的服务端采用多通道视频捕获、异步数据压缩和按需传输的技术,并且具有可伸缩的配置,既能保证服务质量又便于部署.本地的大屏幕客户端提供用户友好的图形界面来对远程的应用进行操控.实验数据表明,系统具备实时交互性能.