Content-Aware Distortion-Fair Video Streaming in Congested Networks

Internet is experiencing a substantial growth of video traffic. Given the limited network bandwidth resources, how to provide Internet users with good video playback quality-of-service (QoS) is a key problem. For video clips competing bandwidth, we propose an approach of Content-Aware distortion-Fair (CAF) video delivery scheme, which is aware of the characteristics of video frames and ensures max-min distortion-fair sharing among video flows. CAF leverages content-awareness to prioritize packet dropping during congestion. Different from bandwidth fair sharing, CAF targets end-to-end video playback quality fairness among users. The proposed CAF approach does not require rate-distortion modeling of the source, which is difficult to estimate. Instead, it exploits the temporal prediction structure of the video sequences along with a frame drop distortion metric to guide resource allocations and coordinations. Experimental results show that the proposed approach operates with limited overhead in computation and communication, and yields better QoS, especially when the network is congested.     

Concealment of Whole-Picture Loss in Hierarchical B-Picture Scalable Video Coding

H.264/AVC scalable video coding (H.264/AVC SVC), as the scalable extension of H.264/AVC, offers the flexible adaptivity in terms of spatial, temporal and SNR scalabilities for the generated bitstream. However, such compressed video still suffers from the bad playback quality when packet loss occurs over unreliable networks. In this paper, we present an error concealment algorithm to tackle the whole-picture loss problem in H.264/AVC SVC when hierarchical B-picture coding is used to support temporal scalability. In the proposed algorithm, by taking advantage of the temporal relationship among the adjacent video pictures, the motion information of the lost picture is derived simply and efficiently based on the principle of temporal direct mode. Utilizing the derived motion information, the lost picture is concealed by performing motion compensation on the correctly received temporally previous and future video pictures. The experimental results demonstrate that as a post-processing tool, the proposed error concealment algorithm is able to significantly improve both the objective and subjective qualities of the decoded video pictures in the presence of packet losses when compared to the error concealment algorithm used in H.264/AVC SVC reference software. The proposed method can also be applied to H.264/AVC with hierarchical B-picture coding for error concealment.     

一种时域和梯度域相结合的视频对象提取算法

为了从复杂背景中精确地提取出视频对象,提出了一种融合时域和梯度域信息的视频对象提取算法,可以有效地提取出复杂背景下的视频运动对象,并解决前景与背景一致情况下,背景剔除方法所带来的空洞问题。首先在时域空间中分别采用背景剔除和帧间差分方法生成初步的视频对象,并利用形态学中的二值腐蚀和膨胀方法对视频对象进行处理;然后,在梯度域空间中用Sobel算子进行视频对象边缘检测,并结合时域空间中的视频对象,生成精确的视频对象轮廓边缘;最后,采用启发式搜索方法连接视频对象轮廓边缘点,进而提取出视频对象。实验结果表明,该方法能够比较完整精确地从复杂背景中提取出视频对象。     

(2+1)D多时空信息融合模型及在行为识别的应用

针对常规的卷积神经网络时空感受野尺度单一,难以提取视频中多变的时空信息的问题,利用（2+1）D模型将时间信息和空间信息在一定程度上解耦的特性,提出了（2+1）D多时空信息融合的卷积残差神经网络,并用于人体行为识别.该模型以3×3空间感受野为主,1×1空间感受野为辅,与3种不同时域感受野交叉组合构建了6种不同尺度的时空感受野.提出的多时空感受野融合模型能够同时获取不同尺度的时空信息,提取更丰富的人体行为特征,因此能够更有效识别不同时间周期、不同动作幅度的人体行为.另外提出了一种视频时序扩充方法,该方法能够同时在空间信息和时间序列扩充视频数据集,丰富训练样本.提出的方法在公共视频人体行为数据集UCF101和HMDB51上子视频的识别率超过或接近最新的视频行为识别方法.     

Video Streaming Over In-Home Power Line Networks

The deployment of power line communication technology for broadband video streaming remains a challenge because power lines are not originally designed for signal transmission. Scalable video is a viable approach that can cope with the bandwidth fluctuation of power line communication networks provided that the bandwidth information is available. In this paper we first investigate how the interference caused by electrical appliances or power supplies affects the power line channel bandwidth and packet transmission. Then we take the obtained characteristics of in-home power line network into account in the design of a simple but effective heuristic-based application-layer bandwidth estimation scheme, for which the cutoff rate is estimated from the packet size and the physical-layer data rates. Experimental results show that the proposed approach can effectively combat the noise interference and deliver robust video streaming over power line.      

Real-time haptic interaction with RGBD video streams

Video interaction is a common way of communication in cyberspace. It can become more immersive by incorporating haptic modality. Using commonly available depth sensing controllers like Microsoft Kinect, information about the depth of a scene can be captured in real-time together with the video. In this paper, we present a method for real-time haptic interaction with videos containing depth data. Forces are computed based on the depth information. Spatial and temporal filtering of the depth stream is used to provide stability of force feedback delivered to the haptic device. Fast collision detection ensures the proposed approach to be used in real-time. We present an analysis of various factors that affect algorithm performance. The usefulness of the approach is illustrated by highlighting possible application scenarios.     

Vector-quantization-based video codec for software-only playback on personal computers

This paper discusses a video compression and decompression method based on vector quantization (VQ) for use on general purpose computer systems without specialized hardware. After describing basic VQ coding, we survey common VQ variations and discuss their impediments in light of the target application. We discuss how the proposed video codec was designed to reduce computational complexity in every principal task of the video codec process. We propose a classified VQ scheme that satisfies the data rate, image quality, decoding speed, and encoding speed objectives for software-only video playback. The functional components of the proposed VQ method are covered in detail. The method employs a pseudo-YUV color space and criteria to detect temporal redundancy and low spatial frequency regions. A treestructured-codebook generation algorithm is proposed to reduce encoding execution time while preserving image quality. Two separate vector codebooks, each generated with the treestructured search, are employed for detail and low spatial frequency blocks. Codebook updating and sharing are proposed to further improve encoder speed and compression.     

基于时序概率超图模型的视频多语义标注

在基于语义的视频检索系统中,为了弥补视频底层特征与高层用户需求之间的差异,提出了时序概率超图模型。它将时间序列因素融入到模型的构建中,在此基础上提出了一种基于时序概率超图模型的视频多语义标注框架（TPH-VMLAF）。该框架结合视频时间相关性,通过使用基于时序概率超图的镜头多标签半监督分类学习算法对视频镜头进行多语义标注。标注过程中同时解决了已标注视频数据不足和多语义标注的问题。实验结果表明,该框架提高了标注的精确度,表现出了良好的性能。     

Social video annotation by combining features with a tri-adaptation approach

Online social video websites such as YouTube allow users to manually annotate their video documents with textual labels. These labels can be used as indexing keywords to facilitate search and organization of video data. However, manual video annotation is usually a labor-intensive and time-consuming process. In this work, we propose a novel social video annotation approach that combines multiple feature sets based on a tri-adaptation approach. For the shots in each video, they are annotated by aggregating models that are learned from three complementary feature sets. Meanwhile, the models are collaboratively adapted by exploring unlabeled shots. In this sense, the method can be viewed as a novel semi-supervised algorithm that explores three complementary views. Our approach also exploits the temporal smoothness of video labels by applying a label correction strategy. Experiments on a web video dataset demonstrate the effectiveness of the proposed approach.     

Adaptive bandwidth allocation based on playback tunnel to support video on demand service over ATM networks

This paper focuses on adaptive bandwidth allocation (ABA) for transporting pre-stored VBR compressed video data across networks in support of video-on-demand (VoD) service. Two key issues are addressed. Firstly, a simplified dynamic programming method based on playback tunnel is proposed to predict the bandwidth which can be dynamically allocated for transporting video segment to guarantee that the playback buffer neither underflows nor overflows. Secondly, ABA schemes with different criteria including middle-tunnel ABA (MT-ABA) and reduced-bandwidth-variability ABA (RBV-ABA) are investigated. The proposed ABA schemes are evaluated by a set of real-life MPEG video traces. The obtained results show that the RBV-ABA scheme outperforms MT-ABA scheme in terms of peak rate, traffic burstiness and bandwidth variability. On the other hand, the MT-ABA scheme is more significant for avoiding the playback buffer overflow or underflow than that of RBV-ABA.     

音视频流实时分离及同步播放系统设计与实现

通过音频与视频解码器和相关的音视频流方案实现上位机与下位机音视频流数据的编码解码及重新合成,并给出了音视频在播放过程中实现同步校正的方法,实现了在有高音质外放设备情况下的上下位机播放的准确同步。系统在Android平台上已获得实现,所使用的音视频实时分离及同步播放方法能成功实现智能终端视频播放和高音质外放音频播放的准确同步,并在上位机成功合成新的音视频文件。     

Multisource surveillance video data coding with hierarchical knowledge library

The rapidly increasing surveillance video data has challenged the existing video coding standards. Even though knowledge based video coding scheme has been proposed to remove redundancy of moving objects across multiple videos and achieved great coding efficiency improvement, it still has difficulties to cope with complicated visual changes of objects resulting from various factors. In this paper, a novel hierarchical knowledge extraction method is proposed. Common knowledge on three coarse-to-fine levels, namely category level, object level and video level, are extracted from history data to model the initial appearance, stable changes and temporal changes respectively for better object representation and redundancy removal. In addition, we apply the extracted hierarchical knowledge to surveillance video coding tasks and establish a hybrid prediction based coding framework. On the one hand, hierarchical knowledge is projected to the image plane to generate reference for I frames to achieve better prediction performance. On the other hand, we develop a transform based prediction for P/B frames to reduce the computational complexity while improve the coding efficiency. Experimental results demonstrate the effectiveness of our proposed method.

     

基于画面组单元时序控制算法的H.264实时流关键帧预处理

网络带宽特别是上行带宽受限情况下,基于H.264高压缩率的网络音视频通话或视频会议普遍会出现不同程度的丢包现象,从而造成流媒体播放的花屏等质量问题,影响视频通话或视频会议的效果。针对上述问题,提出了基于关键帧预处理的实时流媒体播放质量控制方法。该方法采用单元时序控制算法对视频关键坏帧进行实时侦测和取舍,进而减少花屏情况。使用该方法将降低后处理计算的时空成本,并提高流媒体播放的流畅性。通过原始帧播放、后处理播放、关键帧预处理播放3种处理方法的实验对比,证明基于播放单元时序控制算法的实时流媒体播放器,不但明显提高了播放的流畅性,播放后处理的计算复杂度也降低了40%以上。结果表明,该方法对提高播放质量、减少花屏情况有着突出的效果。     

Dual-plan bandwidth smoothing for layer-encoded video

Traditional bandwidth smoothing techniques can be naturally supported by the renegotiated constant bit rate (RCBR) service model, but renegotiation failure in RCBR may cause buffer underflow and interrupt the playback of video. To address this concern, a novel dual-plan bandwidth smoothing (DBS) scheme is proposed in this paper by taking advantage of the SNR scalability of layer-encoded video. Upon renegotiation failure, the proposed scheme can adaptively discard certain enhancement layers to guarantee continuous video playback at the original frame rate. Experiments are carried out to demonstrate the validity of the proposed scheme. The impacts of renegotiation interval, granularity of enhancement layers, and playback buffer size on resulted video quality are also studied. From the simulation results, it is shown that the performance of the RCBR-based DBS scheme can be improved by 1) reducing the minimum time gap of renegotiation interval; 2) employing multilayer video encoding with finer granularity; and/or 3) increasing the playback buffer size.     

一种基于用户播放行为序列的个性化视频推荐策略

本文针对在线视频服务网站的个性化推荐问题,提出了一种基于用户播放行为序列的个性化推荐策略.该策略通过深度神经网络词向量模型分析用户播放视频行为数据,将视频映射成等维度的特征向量,提取视频的语义特征.聚类用户播放历史视频的特征向量,建模用户兴趣分布矩阵.结合用户兴趣偏好和用户观看历史序列生成推荐列表.在大规模的视频服务系统中进行了离线实验,相比随机算法、基于物品的协同过滤和基于用户的协同过滤传统推荐策略,本方法在用户观看视频的Top-N推荐精确率方面平均分别获得22.3%、30.7%和934%的相对提升,在召回率指标上分别获得52.8%、41%和1065%的相对提升.进一步地与矩阵分解算法SVD++、基于双向LSTM模型和注意力机制的Bi-LSTM+Attention算法和基于用户行为序列的深度兴趣网络DIN比较,Top-N推荐精确率和召回率也得到了明显提升.该推荐策略不仅获得了较高的精确率和召回率,还尝试解决传统推荐面临大规模工业数据集时的数据要求严苛、数据稀疏和数据噪声等问题.     

三维卷积和视频帧采样算法下斗殴检测技术

针对监控视频中斗殴行为检测的需求,提出了一种新的基于三维卷积神经网络和视频帧采样算法的斗殴行为检测方法。针对监控视频行为检测起始定位的难点,提出了一种利用基于人体姿态信息的关键区域检测算法定位斗殴行为起始帧的方法,形成了斗殴行为预识别空间。针对深度学习训练数据冗余和优化程度不够的问题,提出了基于时间采样的视频帧采样算法,并且搭建了一个三维卷积神经网络,使网络学习到整个行为动作的时空信息。实验结果证明了所提方法在两个公共数据集上取得了优越的性能。     

A Multimodal Scheme for Program Segmentation and Representation in Broadcast Video Streams

With the advance of digital video recording and playback systems, the request for efficiently managing recorded TV video programs is evident so that users can readily locate and browse their favorite programs. In this paper, we propose a multimodal scheme to segment and represent TV video streams. The scheme aims to recover the temporal and structural characteristics of TV programs with visual, auditory, and textual information. In terms of visual cues, we develop a novel concept named program-oriented informative images (POIM) to identify the candidate points correlated with the boundaries of individual programs. For audio cues, a multiscale Kullback-Leibler (K-L) distance is proposed to locate audio scene changes (ASC), and accordingly ASC is aligned with video scene changes to represent candidate boundaries of programs. In addition, latent semantic analysis (LSA) is adopted to calculate the textual content similarity (TCS) between shots to model the inter-program similarity and intra-program dissimilarity in terms of speech content. Finally, we fuse the multimodal features of POIM, ASC, and TCS to detect the boundaries of programs including individual commercials (spots). Towards effective program guide and attracting content browsing, we propose a multimodal representation of individual programs by using POIM images, key frames, and textual keywords in a summarization manner. Extensive experiments are carried out over an open benchmarking dataset TRECVID 2005 corpus and promising results have been achieved. Compared with the electronic program guide (EPG), our solution provides a more generic approach to determine the exact boundaries of diverse TV programs even including dramatic spots.     

Priority-based Media Delivery using SVC with RTP and HTTP streaming

Media delivery, especially video delivery over mobile channels may be affected by transmission bitrate variations or temporary link interruptions caused by changes in the channel conditions or the wireless interface. In this paper, we present the use of Priority-based Media Delivery (PMD) for Scalable Video Coding (SVC) to overcome link interruptions and channel bitrate reductions in mobile networks by performing a transmission scheduling algorithm that prioritizes media data according to its importance. The proposed approach comprises a priority-based media pre-buffer to overcome periods under reduced connectivity. The PMD algorithm aims to use the same transmission bitrate and overall buffer size as the traditional streaming approach, yet is more likely to overcome interruptions and reduced bitrate periods. PMD achieves longer continuous playback than the traditional approach, avoiding disruptions in the video playout and therefore improving the video playback quality. We analyze the use of SVC with PMD in the traditional RTP streaming and in the adaptive HTTP streaming context. We show benefits of using SVC in terms of received quality during interruption and re-buffering time, i.e. the time required to fill a desired pre-buffer at the receiver. We present a quality optimization approach for PMD and show results for different interruption/bitrate-reduction scenarios.