Rate-distortion optimized distributed packet scheduling of multiple video streams over shared communication resources

We consider the problem of distributed packet selection and scheduling for multiple video streams sharing a communication channel. An optimization framework is proposed, which enables the multiple senders to coordinate their packet transmission schedules, such that the average quality over all video clients is maximized. The framework relies on rate-distortion information that is used to characterize a video packet. This information consists of two quantities: the size of the packet in bits, and its importance for the reconstruction quality of the corresponding stream. A distributed streaming strategy then allows for trading off rate and distortion, not only within a single video stream, but also across different streams. Each of the senders allocates to its own video packets a share of the available bandwidth on the channel in proportion to their importance. We evaluate the performance of the distributed packet scheduling algorithm for two canonical problems in streaming media, namely adaptation to available bandwidth and adaptation to packet loss through prioritized packet retransmissions. Simulation results demonstrate that, for the difficult case of scheduling nonscalably encoded video streams, our framework is very efficient in terms of video quality, both over all streams jointly and also over the individual videos. Compared to a conventional streaming system that does not consider the relative importance of the video packets, the gains in performance range up to 6 dB for the scenario of bandwidth adaptation, and even up to 10 dB for the scenario of random packet loss adaptation.     

Multi-camera tele-immersion system with real-time model driven data compression

Vision-based full-body 3D reconstruction for tele-immersive applications generates large amount of data points, which have to be sent through the network in real time. In this paper, we introduce a skeleton-based compression method using motion estimation where kinematic parameters of the human body are extracted from the point cloud data in each frame. First we address the issues regarding the data capturing and transfer to a remote site for the tele-immersive collaboration. We compare the results of the existing compression methods and the proposed skeleton-based compression technique. We examine the robustness and efficiency of the algorithm through experimental results with our multi-camera tele-immersion system. The proposed skeleton-based method provides high and flexible compression ratios from 50:1 to 5000:1 with reasonable reconstruction quality (peak signal-to-noise ratio from 28 to 31 dB) while preserving real-time (10+ fps) processing.     

Multistreaming of 3-D Scenes With Optimized Transmission and Rendering Scalability

Three-dimensional (3D) graphic scenes require considerable network bandwidth to be transmitted and computing power to be rendered on a user's terminal. Toward high-quality display in real time, we propose a sender-driven mechanism for streaming 3D scenes in a resource-constrained environment. In doing so, objects are encoded into multiresolutions to provide transmission and rendering scalability, and a weighted distortion metric is developed to measure the quality of a scene rendered with multiresolution objects, modeling objects' unequal importance regarding display. To preserve the manipulation independency of multiple objects in data delivery while provide preferential treatment for different objects as well as different layers of each object, transmission of the objects is performed over multiple streams in a partially sequenced and partially reliable fashion. A rate-distortion optimization framework is developed, which determines an optimal level of reliability for every chunk of data in each stream, taking into account the rendering importance of the object, the distortion-rate performance of the data chunks, and the statistics of the network link. Compared with heuristical methods, simulation results show that the proposed framework maximizes the display quality of the scene while minimizing the amount of data that needs to be processed by the client's rendering engine     

基于3维立体技术的视觉加密

视觉加密是新的加密研究领域,其优点是利用人眼视觉系统的特性直接对被加密的内容进行解码,而不需要用专用软件或硬件进行复杂的解密计算。近年来,由于自动立体显示技术和立体眼镜技术的发展,使得3维显示的应用日趋广泛,而传统的视觉加密方法在3维视频领域内尚无法取得令人满意的效果,因此3维立体视觉加密研究具有重要意义。为了更有效地对3维视频进行加密,利用3维显示的原理和特点,将其与视觉加密的优点相结合,提出了一种3维立体视觉加密新方法,该方法利用人眼视觉特性中的视差特性将隐藏信息巧妙地隐藏在右视图中,实验结果表明,该方法是有效的。     

A Distributed Stream Query Optimization Framework through Integrated Planning and Deployment

This paper addresses the problem of optimizing multiple distributed stream queries that are executing simultaneously in distributed data stream systems. We argue that the static query optimization approach of "plan, then deployment” is inadequate for handling distributed queries involving multiple streams and node dynamics faced in distributed data stream systems and applications. Thus, the selection of an optimal execution plan in such dynamic and networked computing systems must consider operator ordering, reuse, network placement, and search space reduction. We propose to use hierarchical network partitions to exploit various opportunities for operator-level reuse while utilizing network characteristics to maintain a manageable search space during query planning and deployment. We develop top-down, bottom-up, and hybrid algorithms for exploiting operator-level reuse through hierarchical network partitions. Formal analysis is presented to establish the bounds on the search space and suboptimality of our algorithms. We have implemented our algorithms in the IFLOW [CHECK END OF SENTENCE] system, an adaptive distributed stream management system. Through simulations and experiments using a prototype deployed on Emulab [CHECK END OF SENTENCE], we demonstrate the effectiveness of our framework and our algorithms.     

Tele-immersive product evaluation: a review and an implementation framework

A novel virtual tele-immersive product evaluation environment is conceived. The components for the tele-immersive virtual environment for product evaluation include a robust virtual reality (VR) hardware system, associated VR driving software, development tool for the tele-immersive virtual environment, networking software, user representation scheme and tools for developing 3D models and incorporating dynamic properties into the models. We have developed a model to allow users to collaboratively evaluate products using the CAVE^TM, Performer, CAVERN, CAVEActors, Pro/ENGINEER, and ADAMS.     

Software architecture design for streaming Java RMI

In recent years, network streaming becomes a highly popular research topic in computer science due to the fact that a large proportion of network traffic is occupied by multimedia streaming. In this paper we present novel methodologies for enhancing the streaming capabilities of Java RMI. Our streaming support for Java RMI includes the pushing mechanism, which allows the servers to push data in a streaming fashion to the client site, and the aggregation mechanism, which allows the client site to make a single remote invocation to gather data from multiple servers that keep replicas of data streams and aggregate partial data into a complete data stream. In addition, our system also allows the client site to forward local data to other clients. Our framework is implemented by extending the Java RMI stub to allow custom designs for streaming buffers and controls, and by providing a continuous buffer for raw data in the transport layer socket. This enhanced framework allows standard Java RMI services to enjoy streaming capabilities. In addition, we propose aggregation algorithms as scheduling methods in such an environment. Preliminary experiments using our framework demonstrate its promising performance in the provision of streaming services in Java RMI layers.     

Uncovering Global Icebergs in Distributed Streams: Results and Implications

Discovering icebergs in distributed streams of data is an important problem for a number of applications in networking and databases. While previous work has concentrated on measuring these icebergs in the non-distributed streaming case or in the non-streaming distributed case, we present a general framework that allows for distributed processing across multiple streams of data. We compare several of the state-of-the-art streaming algorithms for estimating local elephants in the individual streams. However, since an iceberg may be hidden by being distributed across many different streams, we add a sampling component to handle such cases. We provide a novel taxonomy of current sketches and perform a thorough analysis of the strengths and weaknesses of each scheme under various QoS metrics, using both real and synthetic Internet trace data. We summarize their performance and discuss the implications for the future design of sketches.     

Mining frequent itemsets over distributed data streams by continuously maintaining a global synopsis

Mining frequent itemsets over data streams has attracted much research attention in recent years. In the past, we had developed a hash-based approach for mining frequent itemsets over a single data stream. In this paper, we extend that approach to mine global frequent itemsets from a collection of data streams distributed at distinct remote sites. To speed up the mining process, we make the first attempt to address a new problem on continuously maintaining a global synopsis for the union of all the distributed streams. The mining results therefore can be yielded on demand by directly processing the maintained global synopsis. Instead of collecting and processing all the data in a central server, which may waste the computation resources of remote sites, distributed computations over the data streams are performed. A distributed computation framework is proposed in this paper, including two communication strategies and one merging operation. These communication strategies are designed according to an accuracy guarantee of the mining results, determining when and what the remote sites should transmit to the central server (named coordinator). On the other hand, the merging operation is exploited to merge the information received from the remote sites into the global synopsis maintained at the coordinator. By the strategies and operation, the goal of continuously maintaining the global synopsis can be achieved. Rooted in the continuously maintained global synopsis, we propose a mining algorithm for finding global frequent itemsets. Moreover, the correctness guarantees of the communication strategies and merging operation, and the accuracy guarantee analysis of the mining algorithm are provided. Finally, a series of experiments on synthetic datasets and a real dataset are performed to show the effectiveness and efficiency of the distributed computation framework.     

A system for real-time panorama generation and display in tele-immersive applications

Wide field-of-view (FOV) is necessary for many industrial applications, such as air traffic control, large vehicle driving and navigation. Unfortunately, the supporting structure/frame in most systems usually blocks part of the view, results in "blind spot" and raises the risk to the pilot. In this paper, we introduce a video-based tele-immersive system, called the immersive cockpit. It captures live videos from the working site and recreates an immersive environment at the remote site where the pilot situates. It immerses the pilot at the remote site with a panoramic view of the environment, and hence improves interactivity and safety. The design goals of our system are real-time, live, low-cost, and scalable. We stitch multiple video streams captured from ordinary charged couple device cameras to generate a panoramic video. To avoid being blocked by the supporting frame, we allow a flexible placement of cameras. This approach trades the accuracy of the generated panoramic image for a larger FOV. To reduce the computation, parameters for stitching are determined once during the system initialization. The panoramic video is presented on an immersive display which covers the FOV of the viewer. We discuss how to correctly present the panoramic video on this nonplanar immersive display screen by sweet spot relocation. We also present the result and the performance evaluation of the system.     

A scalable bandwidth guaranteed distributed continuous media file system using network attached autonomous disks

Repository for continuous media data differs from that of the traditional text-based data both in storage space and streaming bandwidth requirements. The file systems used for continuous media streams need to support large volumes and high bandwidth. We propose a scalable distributed continuous media file system built using autonomous disks. Autonomous disks are attached directly to the network and are able to perform lightweight processing. We discuss different ways to realize the autonomous disk, and describe a prototype implementation on a Linux platform using PC-based hardware. We present the basic requirements of the continuous media file system and present the design methodology and a prototype Linux-based implementation of the distributed file system that supports the requirements. We present experimental results on the performance of the proposed file system prototyped using autonomous disks. We show that the performance of the file system scales linearly with the number of disks and the number of clients. The file system performs much superior to NFS running on the same hardware platform and can deliver higher raw disk bandwidth to the applications. We also present bandwidth and time sensitive read/write procedures for the file system and show that the file system can provide strict bandwidth guarantees for continuous media streams.     

Interactive multiscale tensor reconstruction for multiresolution volume visualization

Large scale and structurally complex volume datasets from high-resolution 3D imaging devices or computational simulations pose a number of technical challenges for interactive visual analysis. In this paper, we present the first integration of a multiscale volume representation based on tensor approximation within a GPU-accelerated out-of-core multiresolution rendering framework. Specific contributions include (a) a hierarchical brick-tensor decomposition approach for pre-processing large volume data, (b) a GPU accelerated tensor reconstruction implementation exploiting CUDA capabilities, and (c) an effective tensor-specific quantization strategy for reducing data transfer bandwidth and out-of-core memory footprint. Our multiscale representation allows for the extraction, analysis and display of structural features at variable spatial scales, while adaptive level-of-detail rendering methods make it possible to interactively explore large datasets within a constrained memory footprint. The quality and performance of our prototype system is evaluated on large structurally complex datasets, including gigabyte-sized micro-tomographic volumes.     

Community Streaming With Interactive Visual Overlays: System and Optimization

Community streaming is an enhanced form of joint content viewing where a sense of community is reinforced by the addition of interactive visual overlays, controlled in real-time by viewers, on top of a shared video stream. As a concrete example, we describe a community video system called ECHO, where personalized avatars are overlaid on top of a real-time encoded video stream of an Internet game for multicast consumption. Recognizing that only the visual overlays are generated live, we propose schemes that encode and schedule the live and non-live portions of the overlaid video separately in order to exploit the difference in delay sensitivity of the two, leading to video streams that contain two sub-streams with different delay constraints. We show that, in the known channel case, a low complexity ldquoearliest deadline firstrdquo packet scheduling algorithm minimizes receiver buffer delay. We also analyze the case where multiple streams are multiplexed, which allows us to quantify the potential gains of allowing different delay constraints for different sub-streams. We show that a ldquowater fillingrdquo strategy maximizes the total number of streams that can be supported. Simulation results show that the bandwidth necessary to maintain low-latency for visual overlays is reduced by about 40% when our proposed sub-stream approach is used. For multiplexing of multiple streams, our approach can increase the number of supported streams (e.g., a 30% increase when around ten streams are multiplexed).     

An effective 3D target recognition model imitating robust methods of the human visual system

This paper presents a model of 3D object recognition motivated from the robust properties of human vision system (HVS). The HVS shows the best efficiency and robustness for an object identification task. The robust properties of the HVS are visual attention, contrast mechanism, feature binding, multi-resolution, size tuning, and part-based representation. In addition, bottom-up and top-down information are combined cooperatively. Based on these facts, a plausible computational model integrating these facts under the Monte Carlo optimization technique was proposed. In this scheme, object recognition is regarded as a parameter optimization problem. The bottom-up process is used to initialize parameters in a discriminative way; the top-down process is used to optimize them in a generative way. Experimental results show that the proposed recognition model is feasible for 3D object identification and pose estimation in visible and infrared band images.     

Scalable 3D video of dynamic scenes

In this paper we present a scalable 3D video framework for capturing and rendering dynamic scenes. The acquisition system is based on multiple sparsely placed 3D video bricks, each comprising a projector, two grayscale cameras, and a color camera. Relying on structured light with complementary patterns, texture images and pattern-augmented views of the scene are acquired simultaneously by time-multiplexed projections and synchronized camera exposures. Using space–time stereo on the acquired pattern images, high-quality depth maps are extracted, whose corresponding surface samples are merged into a view-independent, point-based 3D data structure. This representation allows for effective photo-consistency enforcement and outlier removal, leading to a significant decrease of visual artifacts and a high resulting rendering quality using EWA volume splatting. Our framework and its view-independent representation allow for simple and straightforward editing of 3D video. In order to demonstrate its flexibility, we show compositing techniques and spatiotemporal effects.     

Dynamic Resource Allocation for MGS H.264/AVC Video Transmission Over Link-Adaptive Networks

In this paper, we address the problem of efficiently allocating network resources to support multiple scalable video streams over a constrained wireless channel. We present a resource allocation framework that jointly optimizes the operation of the link adaptation scheme in the physical layer (PHY), and that of a traffic control module in the network or medium access control (MAC) layer in multirate wireless networks, while satisfying bandwidth/capacity constraints. Multirate networks, such as IEEE 802.16 or IEEE 802.11, adjust the PHY coding and modulation schemes to maintain the reliability of transmission under varying channel conditions. Higher reliability is achieved at the cost of reduced PHY bit-rate which in turn necessitates a reduction in video stream bit-rates. The rate reduction for scalable video is implemented using a traffic control module. Conventional solutions operate unaware of the importance and loss tolerance of data and drop the higher layers of scalable video altogether. In this paper, we consider medium grain scalable (MGS) extension of H.264/AVC video and develop new rate and distortion models that characterize the coded bitstream. Performance evaluations show that our proposed framework results in significant gains over existing schemes in terms of average video PSNR that can reach 3 dB in some cases for different channel SNRs and different bandwidth budgets.      

Render2MPEG: A Perception-based Framework Towards Integrating Rendering and Video Compression

Currently 3D animation rendering and video compression are completely independent processes even if rendered frames are streamed on‐the‐fly within a client‐server platform. In such scenario, which may involve time‐varying transmission bandwidths and different display characteristics at the client side, dynamic adjustment of the rendering quality to such requirements can lead to a better use of server resources. In this work, we present a framework where the renderer and MPEG codec are coupled through a straightforward interface that provides precise motion vectors from the rendering side to the codec and perceptual error thresholds for each pixel in the opposite direction. The perceptual error thresholds take into account bandwidth‐dependent quantization errors resulting from the lossy com‐pression as well as image content‐dependent luminance and spatial contrast masking. The availability of the discrete cosine transform (DCT) coefficients at the codec side enables to use advanced models of the human visual system (HVS) in the perceptual error threshold derivation without incurring any significant cost. Those error thresholds are then used to control the rendering quality and make it well aligned with the compressed stream quality. In our prototype system we use the lightcuts technique developed by Walter et al., which we enhance to handle dynamic image sequences, and an MPEG‐2 implementation. Our results clearly demonstrate many advantages of coupling the rendering with video compression in terms of faster rendering. Furthermore, temporally coherent rendering leads to a reduction of temporal artifacts.     

Bottom-up spatiotemporal visual attention model for video analysis

The human visual system (HVS) has the ability to fixate quickly on the most informative (salient) regions of a scene and therefore reducing the inherent visual uncertainty. Computational visual attention (VA) schemes have been proposed to account for this important characteristic of the HVS. A video analysis framework based on a spatiotemporal VA model is presented. A novel scheme has been proposed for generating saliency in video sequences by taking into account both the spatial extent and dynamic evolution of regions. To achieve this goal, a common, image-oriented computational model of saliency-based visual attention is extended to handle spatiotemporal analysis of video in a volumetric framework. The main claim is that attention acts as an efficient preprocessing step to obtain a compact representation of the visual content in the form of salient events/objects. The model has been implemented, and qualitative as well as quantitative examples illustrating its performance are shown.     

DIASPORA: A highly distributed web-query processing system

Current proposals for web querying systems have assumed a centralized processing architecture wherein data is shipped from the remote sites to the user's site. We present here the design and implementation of DIASPORA, a highly distributed query processing system for the web. It is based on the premise that several web applications are more naturally processed in a distributed manner, opening up possibilities of significant reductions in network traffic and user response times. DIASPORA is built over an expressive graph-based data model that utilizes simple heuristics and lends itself to automatic generation. The model captures both the content of web documents and the hyperlink structural framework of a web site. Distributed queries on the model are expressed through a declarative language that permits users to explicitly specify navigation. DIASPORA implements a query-shipping model wherein queries are autonomously forwarded from one web-site to another, without requiring much coordination from the query originating site. Its design addresses a variety of interesting issues that arise in the distributed web context including determining query completion, handling query rewriting, supporting query termination and preventing multiple computations of a query at a site due to the same query arriving through different paths in the hyperlink framework. The DIASPORA system is currently operational and is undergoing testing on our campus network. In this paper we describe the design of the system and report initial performance results that indicate significant performance improvements over comparable centralized approaches. This revised version was published online in August 2006 with corrections to the Cover Date.