Inter-destination multimedia synchronization: schemes, use cases and standardization

Traditionally, the media consumption model has been a passive and isolated activity. However, the advent of media streaming technologies, interactive social applications, and synchronous communications, as well as the convergence between these three developments, point to an evolution towards dynamic shared media experiences. In this new model, geographically distributed groups of consumers, independently of their location and the nature of their end-devices, can be immersed in a common virtual networked environment in which they can share multimedia services, interact and collaborate in real-time within the context of simultaneous media content consumption. In most of these multimedia services and applications, apart from the well-known intra and inter-stream synchronization techniques that are important inside the consumers’ playout devices, also the synchronization of the playout processes between several distributed receivers, known as multipoint, group or Inter-destination multimedia synchronization (IDMS), becomes essential. Due to the increasing popularity of social networking, this type of multimedia synchronization has gained in popularity in recent years. Although Social TV is perhaps the most prominent use case in which IDMS is useful, in this paper we present up to 19 use cases for IDMS, each one having its own synchronization requirements. Different approaches used in the (recent) past by researchers to achieve IDMS are described and compared. As further proof of the significance of IDMS nowadays, relevant organizations’ (such as ETSI TISPAN and IETF AVTCORE Group) efforts on IDMS standardization (in which authors have been and are participating actively), defining architectures and protocols, are summarized.     

Smooth Control of Adaptive Media Playout for Video Streaming

Client-side data buffering is a common technique to deal with media playout interruptions of streaming video caused by network jitters and packet losses of best-effort networks. However, stronger playout interruption protection inevitably amounts to larger data buffering and results in more memory requirements and longer playout delay. Adaptive media playout (AMP), also a client-side technique, can reduce the buffer requirement and avoid buffer outage but at the expense of visual quality degradation because of the fluctuation of playout speed. In this paper, we propose a novel AMP scheme to keep the video playout as smooth as possible while adapting to the channel condition. The triggering of the playout control is based on buffer variation rather than buffer fullness. Experimental results show that our AMP scheme surpasses conventional schemes in unfriendly network conditions. Unlike previous schemes that are tuned for a specific range of packet loss and network instability, the proposed AMP scheme maintains consistent performance across a wide range of network conditions.     

Adaptive playout scheduling algorithm tailored for real-time packet-based voice conversations over wireless ad-hoc networks

The effective provision of real-time, packet-based voice conversations over multi-hop wireless ad-hoc networks faces several stringent constraints not found in conventional packet-based networks. Indeed, MANETs (mobile ad-hoc networks) are characterized by mobility of all nodes, bandwidth-limited channel, unreliable wireless transmission medium, etc. This environment will surely induce a high delay variation and packet loss rate impairing dramatically the user experienced quality of conversational services such as VoIP. Indeed, such services require the reception of each media unit before its deadline to guarantee a synchronous playback process. This requirement is typically achieved by artificially delaying received packets inside a de-jitter buffer. To enhance the perceptual quality the buffering delay should be adjusted dynamically throughout the vocal conversation.In this work, we describe the design of a playout algorithm tailored for real-time, packet-based voice conversations delivered over multi-hop wireless ad-hoc networks. The designed playout algorithm, which is denoted MAPA (mobility aware playout algorithm), adjusts the playout delay according to node mobility, which characterizes mobile ad-hoc networks, and talk-spurt, which is an intrinsic feature of voice signals. The detection of mobility is done in service passively at the receiver using several metrics gathered at the application layer. The perceptual quality is estimated using an augmented assessment approach relying on the ITU-T E-Model paradigm while including the time varying impairments observed by users throughout a packet-based voice conversation. Simulation results show that the tailored playout algorithm significantly outperforms conventional playout algorithms, specifically over a MANET with a high degree of mobility.     

Arrival process-controlled adaptive media playout with multiple thresholds for video streaming

To improve the playout quality of video streaming services, an arrival process-controlled adaptive media playout (AMP) mechanism is designed in this study. The proposed AMP scheme sets three threshold values, denoted by P _n , L and H, for the playout controller to start playback and dynamically adjust the playout rate based on the buffer fullness. In the preroll period, the playout can start only when the buffer fullness n is not less than the dynamic playback threshold P _n ,?which is determined by the jitters of incoming video frames. In the playback period, if the buffer fullness is below L or over H,?the playout rate will slow down or speed up in a quadratic manner. Otherwise, the playback speed depends on the instantaneous frame arrival rate, which is estimated by the proposed arrival process tracking algorithm. We employ computer simulations to demonstrate the performance of the proposed AMP scheme, and compare it with several conventional AMP mechanisms. Numerical results show that our AMP design can shorten the playout delay and reduce both buffer underflow and overflow probabilities. In addition, our proposed AMP also outperforms traditional AMP schemes in terms of the variance of distortion of playout and the playout curve. Hence, the proposed arrival process-controlled AMP is really an outstanding design.     

Joint optimization of continuity and quality for streaming video

This paper aims to reduce the amount of prebuffering required to ensure a maximum video continuity in streaming. Current approaches do this by slowing the playout frame rate of the decoder, this is known as adaptive media playout (AMP). However, doing this introduces playout distortion to the viewers as the video is played slower than its natural playout rate. We approach this by proposing a frame rate control scheme that jointly adjusts the encoder frame generation rate of the encoder and the playout frame rate of the decoder. In addition to using AMP to improve video continuity, we also allow the encoder to increase the encoder frame generation rate. This means the encoder will be sending more frames to the decoder to quickly increase the number of frames available at the playback buffer, thus lowering the chance of buffer underflow which causes discontinuity in video playback. At the same time, the increase in the number of frames at the playback buffer may mean that the decoder does not need to use AMP to delay the playback, thus lowering the playback distortion. However, the increase in encoder frame generation rate comes at a price because frame quality will need to decrease in order to meet the constraint on available network bandwidth. This implies that the scheme needs to find the optimal trade-off between frame quality, playout distortion and video continuity. To do that, we characterize the frame rate control problem using Lyapunov optimization. We then systematically derive the optimization policies. We also show that these policies can be decoupled into separate encoder and decoder optimization policies, thus allowing for a distributed implementation. Simulation results show significant reductions in the prebuffering requirements over a scheme that perform no frame rate control and lower playout distortions compared to the AMP schemes, while exhibiting a modest drop in frame quality.     

基于PID控制的自适应媒体播放算法

针对视频播放中因缓冲区下溢带来的抖动问题,提出一种基于PID控制的自适应播放算法。算法结合PID控制和自适应播放算法,根据网络信道状态和缓冲区状态控制视频播放速率,并对播放速率的范围和相邻帧播放速率的突变进行控制,在减少缓冲区下溢概率的同时实现视频的平滑播放。该算法在减小缓冲区下溢概率、实现视频的平滑播放以及减少播放延迟方面均优于固定因子自适应播放算法。     

基于概率估计的自适应媒体播放算法

针对缓冲区下溢造成的视频播放抖动和中断问题,提出一种基于概率估计的自适应媒体播放算法。根据网络信道状态和估计的下溢概率和上溢概率控制视频帧的持续播放时间,适当控制播放速率的变化范围和变化量,减少缓冲区下溢概率和播放时延,实现视频平滑播放。仿真结果证明,该算法性能优于传统的自适应媒体播放算法。     

Joint power-playout control for media streaming over wireless links

Media streaming applications over wireless links face various challenges, due to both the nature of the wireless channel and the stringent delivery requirements of media traffic. In this paper, we seek to improve the performance of media streaming over an interference-limited wireless link, by using appropriate transmission and playout control. In particular, we choose both the power at the transmitter and the playout scheduling at the receiver, so as to minimize the power consumption and maximize the media playout quality. We formulate the problem using a dynamic programming approach, and study the structural properties of the optimal solution. We further develop a justified, low-complexity heuristic that achieves significant performance gain over benchmark systems. In particular, our joint power-playout heuristic outperforms: 1) the optimal power control policy in the regime where power is most important and 2) the optimal playout control policy in the regime where media (playout) quality is most important; furthermore, this heuristic has only a slight performance loss as compared to the optimal joint power-playout control policy over the entire range of the investigation.     

Optimize adaptive media playout using dynamic fuzzy logic control for video streaming

Multimedia Tools and Applications - Adaptive Media Playout (AMP) controls adapt playout rate to prevent buffer outage and to reduce delay in playout. Most AMP techniques use buffer fullness or its...     

Model-Based Clustering and Visualization of Navigation Patterns on a Web Site

We present a new methodology for exploring and analyzing navigation patterns on a web site. The patterns that can be analyzed consist of sequences of URL categories traversed by users. In our approach, we first partition site users into clusters such that users with similar navigation paths through the site are placed into the same cluster. Then, for each cluster, we display these paths for users within that cluster. The clustering approach we employ is model-based (as opposed to distance-based) and partitions users according to the order in which they request web pages. In particular, we cluster users by learning a mixture of first-order Markov models using the Expectation-Maximization algorithm. The runtime of our algorithm scales linearly with the number of clusters and with the size of the data; and our implementation easily handles hundreds of thousands of user sessions in memory. In the paper, we describe the details of our method and a visualization tool based on it called WebCANVAS. We illustrate the use of our approach on user-traffic data from msnbc.com.     

Continuous-media courseware server: a study of client interactions

Considerable research has gone into investigating networking and operating system mechanisms to support the transfer and playout of stored continuous media, but there is very little information available about how users actually interact with such systems. Developing a user workload characterization can help in the design and evaluation of efficient CM resource allocation and access mechanisms. The authors developed an interactive Web-based, multimedia, client-server application, known as the Multimedia Asynchronous Networked Individualized Courseware, or MANIC, which streams synchronized CM (currently audio) and HTML documents to remote users. This article presents empirical and analytic characterizations of observed user session-level behavior (for example, the length of individual sessions) and interactive behavior (for example, the time between starting, stopping, and pausing the audio within a session). The data come from a full-semester senior-level course given by the University of Massachusetts to more than 200 students who used MANIC to listen to the stored audio lectures and to view the lecture notes     

On the impact of playout scheduling on the performance of peer-to-peer live streaming

In this paper we examine the impact of the adopted playout policy on the performance of P2P live streaming systems. We argue and demonstrate experimentally that (popular) playout policies which permit the divergence of the playout points of different nodes can deteriorate drastically the performance of P2P live streaming. Consequently, we argue in favor of keeping different playout points “near-in-time”, even if this requires sacrificing (dropping) some late frames that could otherwise be rendered (assuming no strict bidirectional interactivity requirements are in place). Such nearly synchronized playout policies create “positive correlation” with respect to the available frames at different playout buffers. Therefore, they increase the number of upstream relay nodes from which a node can pull frames and thus boost the playout quality of both single-parent (tree) and multiple-parent (mesh) systems. On the contrary, diverging playout points reduce the number of upstream parents that can offer a gapless relay of the stream. This is clearly undesirable and should be avoided as it contradicts the fundamental philosophy of P2P systems which is to supplement an original service point with as many additional ones presented by the very own users of the service.     

Content-Aware Playout and Packet Scheduling for Video Streaming Over Wireless Links

Media streaming over wireless links is a challenging problem due to both the unreliable, time-varying nature of the wireless channel and the stringent delivery requirements of media traffic. In this paper, we use joint control of packet scheduling at the transmitter and content-aware playout at the receiver, so as to maximize the quality of media streaming over a wireless link. Our contributions are twofold. First, we formulate and study the problem of joint scheduling and playout control in the framework of Markov decision processes. Second, we propose a novel content-aware adaptive playout control, that takes into account the content of a video sequence, and in particular the motion characteristics of different scenes. We find that the joint scheduling and playout control can significantly improve the quality of the received video, at the expense of only a small amount of playout slowdown. Furthermore, the content-aware adaptive playout places the slowdown preferentially in the low-motion scenes, where its perceived effect is lower.      

Content-Aware Adaptive Media Playout Controls for Wireless Video Streaming

Video streaming is one of the killer applications for cellular communications. The MPEG-4 fine-granularity scalability video coding technique can adapt to bandwidth variation and random packet errors. In this paper, to explore the impacts of cellular channel characteristics on the tolerance of buffer performance and quality of service, a novel statistical model-based adaptive media playout (AMP) is proposed by utilizing the statistical assumptions of both arrival and departure processes for a better decision on the dynamic threshold adjustment and frame-rate adjustment. Based on third-generation cellular transmission environment, simulation results will demonstrate that as compared to other AMP schemes, the proposed AMP control provides better visual quality with lower complexity.     

Delay reduction techniques for playout buffering

Receiver synchronization of continuous media streams is required to deal with delay differences and variations resulting from delivery over packet networks such as the Internet. This function is commonly provided using per-stream playout buffers which introduce additional delay in order to produce a playout schedule which meets the synchronization requirements. Packets which arrive after their scheduled playout time are considered late and are discarded. In this paper, we present the Concord algorithm, which provides a delay-sensitive solution for playout buffering. It records historical information and uses it to make short-term predictions about network delay with the aim of not reacting too quickly to short-lived delay variations. This allows an application-controlled tradeoff of packet lateness against buffering delay, suitable for applications which demand low delay but can tolerate or conceal a small amount of late packets. We present a selection of results from an extensive evaluation of Concord using Internet traffic traces. We explore the use of aging techniques to improve the effectiveness of the historical information and hence, the delay predictions. The results show that Concord can produce significant reductions in buffering delay and delay variations at the expense of packet lateness values of less than 1%     

Optimized camera handover scheme in free-viewpoint video streaming

Free-viewpoint video (FVV) is a promising approach that allows users to control their viewpoint and generate virtual views from any desired perspective. The individual user viewpoints are synthesized from two or more camera streams and correspondent depth sequences. In case of continuous viewpoint changes, the camera inputs of the view-synthesis process must be changed in a seamless way, to avoid the starvation of the viewpoint synthesizer algorithm. Starvation occurs when the desired user viewpoint cannot be synthesized with the currently streamed camera views, and thus, the FVV playout interrupts. In this paper, we propose three different camera handover schemes (TCC, MA, and SA) based on viewpoint prediction to minimize the probability of playout stalls and find the trade-off between the image quality and the camera handover frequency. Our simulation results show that the introduced camera switching methods can reduce the handover frequency with more than 40%, and hence, the viewpoint synthesis starvation and the playout interruption can be minimized. By providing seamless viewpoint changes, the quality of experience can be significantly improved, making the new FVV service more attractive in the future.     

QoS Filtering and Resource Reservation in an Internet Environment

Multimedia group applications often operate in an environment where the various participants are located on systems and communication links with different capabilities. Mechanisms are required that ensure full-quality media for high-performance workstations but lower-quality media for playout at low-end systems. QoS filters have been proposed as a way to adapt QoS to the user specified level by changing the structure of a media stream in a well defined way. Resource reservation and QoS filter instantiation should be closely integrated since both represent one particular aspect of the provision of individualistic QoS for heterogeneous users in multipeer communications. The Internet reservation protocol RSVP is receiver oriented and allows each receiver to specify its resource requirements. However, no actual mechanisms are defined that adapt the data stream to the receiver specified QoS requirements.In this paper we present an enhanced version of RSVP (called RSVP++) that integrates resource reservation and QoS filter control. In order to achieve this integration we extend the RSVP functional model and define a new QoS service class. RSVP++ can coexist with common RSVP systems, thus, openness and interoperability of the system are ensured.     

Storage allocation policies for time-dependent multimedia data

Multimedia computing requires support for heterogeneous data types with differing storage, communication and delivery requirements. Continuous media data types, such as audio and video, impose delivery requirements that are not satisfied by conventional physical storage organizations. In this paper, we describe a physical organization for multimedia data based on the need to support the delivery of multiple playout sessions from a single rotating-disk storage device. Our model relates disk characteristics to the different media recording and playback rates and derives their storage pattern. This storage organization guarantees that as long as a multimedia delivery process is running, starvation will never occur. Furthermore, we derive bandwidth and buffer constraints for disk access and present an approach to minimize latencies for non-continuous media stored on the same device. The analysis and numerical results indicate the feasibility of using conventional rotating magnetic disk storage devices to support multiple sessions for video-on-demand applications     

TimeArcs: Visualizing Fluctuations in Dynamic Networks

In this paper we introduce TimeArcs, a novel visualization technique for representing dynamic relationships between entities in a network. Force‐directed layouts provide a way to highlight related entities by positioning them near to each other Entities are brought closer to each other (forming clusters) by forces applied on nodes and connections between nodes. In many application domains, relationships between entities are not temporally stable, which means that cluster structures and cluster memberships also may vary across time. Our approach merges multiple force‐directed layouts at different time points into a single comprehensive visualization that provides a big picture overview of the most significant clusters within a user‐defined period of time. TimeArcs also supports a range of interactive features, such as allowing users to drill‐down in order to see details about a particular cluster. To highlight the benefits of this technique, we demonstrate its application to various datasets, including the IMDB co‐star network, a dataset showing conflicting evidences within biomedical literature of protein interactions, and collocated popular phrases obtained from political blogs.     

Augmenting Multi-Party Face-to-Face Interactions Amongst Strangers with User Generated Content

We present the results of an investigation into the role of curated representations of self, which we term Digital Selfs, in augmented multi-party face-to-face interactions. Advancements in wearable technologies (such as Head-Mounted Displays) have renewed interest in augmenting face-to-face interaction with digital content. However, existing work focuses on algorithmic matching between users, based on data-mining shared interests from individuals’ social media accounts, which can cause information that might be inappropriate or irrelevant to be disclosed to others. An alternative approach is to allow users to manually curate the digital augmentation they wish to present to others, allowing users to present those aspects of self that are most important to them and avoid undesired disclosure. Through interviews, video analysis, questionnaires and device logging, of 23 participants in 6 multi-party gatherings where individuals were allowed to freely mix, we identified how users created Digital Selfs from media largely outside existing social media accounts, and how Digital Selfs presented through HMDs were employed in multi-party interactions, playing key roles in facilitating strangers to interact with each other. We present guidance for the design of future multi-party digital augmentations in collaborative scenarios.