移动互联网视频传输优化的架构研究

随着3G移动互联网的快速发展,在手机等移动终端上看视频成为一种日常应用.但互联网上的大部分视频对于移动用户而言,其码率相对较大,而且移动用户的可用带宽受环境影响变化大,不稳定,影响了用户观看视频的体验.提出一种针对移动互联网的视频传输优化解决方案的系统架构,该架构从缓存和压缩两个方面入手,通过缓存缓解运营商的骨干网流量压力,通过视频压缩降低视频码率以满足用户的实际接入带宽.该架构既为运营商节约了带宽、降低了运营成本,同时也保证用户观看视频的连续性,提高用户体验.     

Energy optimization for mobile video streaming via an aggregate model

Wireless video streaming on smartphones drains a significantly large fraction of battery energy, which is primarily consumed by wireless network interfaces for downloading unused data and repeatedly switching radio interface. In this paper, we propose an energy-efficient download scheduling algorithm for video streaming based on an aggregate model that utilizes user’s video viewing history to predict user behavior when watching a new video, thereby minimizing wasted energy when streaming over wireless network interfaces. The aggregate model is constructed by a personal retention model with users’ personal viewing history and the audience retention on crowd-sourced viewing history, which can accurately predict the user behavior of watching videos by balancing “user interest” and “video attractiveness”. We evaluate different users streaming multiple videos in various wireless environments and the results illustrate that the aggregate model can help reduce energy waste by 20 % on average. In addition, we also discuss implementation details and extensions, such as dynamically updating personal retention, balancing audience and personal retention, categorizing videos for accurate model.     

CStream: neighborhood bandwidth aggregation for better video streaming

Despite the popularity of watching videos online, challenges still remain in video streaming in many scenarios. Limited home broadband and mobile phone 3G bandwidths mean many users stream videos at compromised quality. To provide additional bandwidth for streaming, we propose CStream, a system that aggregates bandwidth from multiple cooperating users in a neighborhood environment for better video streaming. CStream exploits the fact that wireless devices have multiple network interfaces and connects cooperating users with a wireless ad-hoc network to aggregate their unused downlink Internet bandwidth. CStream dynamically generates a streaming plan to stream a single video using multiple connections, continuously adapting to changes in the neighborhood and variations in the available bandwidth. CStream is developed and evaluated on a test bed of computers, allowing for a detailed, controlled evaluation of performance. Analysis of the results shows a linear increase in throughput over single-connection streaming and improved video quality as the number of cooperating users in a neighborhood increase.     

Real-time panoramic video streaming system with overlaid interface concept for social media

In this paper, we propose a real-time panoramic video streaming system with overlaid interface concept for social media. The proposed system provides real-time panorama images for smart displays such as smart TVs, smart phones, and tablet PCs. Panorama images are collected at sporting events via panorama cameras. Contents thus collected are sent to servers and then provided to smart devices via live sports video streams. Users select a panorama camera and choose their viewing angle and zooming factor for the selected panorama camera. The proposed system provides immersive and realistic views of live sporting events on users’ displays. Furthermore, an overlaid panoramic interface concept is proposed for immersive live baseball watching combined with tightly integrated social media experience.     

Exploring sharing patterns for video recommendation on YouTube-like social media

YouTube-like video sharing sites (VSSes) have gained increasing popularity in recent years. Meanwhile, Face-book-like online social networks (OSNs) have seen their tremendous success in connecting people of common interests. These two new generation of networked services are now bridged in that many users of OSNs share video contents originating from VSSes with their friends, and it has been shown that a significant portion of views of VSS videos are attributed to this sharing scheme of social networks. To understand how the video sharing behavior, which is largely based on social relationship, impacts users’ viewing pattern, we have conducted a long-term measurement with RenRen and YouKu, the largest online social network and the largest video sharing site in China, respectively. We show that social friends have higher common interest and their sharing behaviors provide guidance to enhance recommended video lists. In this paper, we take a first step toward learning OSN video sharing patterns for video recommendation. An autoencoder model is developed to learn the social similarity of different videos in terms of their sharing in OSNs. We, therefore, propose a similarity-based strategy to enhance video recommendation for YouTube-like social media. Evaluation results demonstrate that this strategy can remarkably improve the precision and recall of recommendations, as compared to other widely adopted strategies without social information.     

一种基于P2P的短视频分享系统

现有在线短视频分享策略通常采用C/S架构,给视频服务器带来较大的带宽压力。为此,提出一种采用点对点方式的在线短视频分享系统IShare,该系统结合用户点播偏好和视频文件之间的社会网络特性实现视频分享。IShare主要包括基于点播兴趣的节点分簇和视频数据源节点的查找2个核心技术。实验结果表明,IShare具备较好的视频数据源节点查找能力,可降低视频服务器带宽资源消耗。     

基于弹幕情感分析和聚类算法的视频用户群体分类

随着数字媒体等技术的发展,出现了弹幕系统这种新型的评论模式并逐渐流行。它能够使视频观众即时发布关于视频情节内容的评论,也可以帮助观众理解视频内容。弹幕文本数据的产生,为短文本处理和实时数据处理提供了新的素材。研究弹幕数据的特点和其表达的情感,可以帮助我们更好地理解视频情节;研究弹幕内容之间的相似度进而分析用户之间的关联关系,不仅能够深入了解弹幕用户的特点、发掘不同视频之间的潜在联系,而且可以为视频制作时受众群体的选择提供更为准确的解决方案。首先将弹幕文本数据进行收集和预处理,然后计算这些文本的情感值。针对弹幕文本口语化的特点,建立了网络弹幕常用词词典。通过改进传统的k-means聚类算法,对所有发表弹幕的用户进行基于情感值的分类。这样的分类可以帮助我们了解观看特定类型视频的观众在情感上的异同点。     

Watching a Movie Alone yet Together: Understanding Reasons for Watching Danmaku Videos

By overlaying timeline-synchronized user comments on videos, Danmaku commenting creates a unique co-viewing experience of online videos. This study aims to understand the reasons for watching or not watching Danmaku videos. From a review of the literature and a pilot study, an initial pool of motivations and hindrances to Danmaku video viewing was gathered. Then, a survey involving 248 participants to identify the underlying factor structures of motivations and hindrances was conducted. Their influences on users’ attitude and behaviors with Danmaku videos were also examined. The results showed that people viewed Danmaku videos to obtain information, entertainment, and social connectedness. Introverted young men with high openness to new experience are more likely to view Danmaku videos. Infrequent viewers refused to watch Danmaku videos mainly because of the visual clutter that resulted from Danmaku comments.     

A Web video retrieval method using hierarchical structure of Web video groups

In this paper, we propose a Web video retrieval method that uses hierarchical structure of Web video groups. Existing retrieval systems require users to input suitable queries that identify the desired contents in order to accurately retrieve Web videos; however, the proposed method enables retrieval of the desired Web videos even if users cannot input the suitable queries. Specifically, we first select representative Web videos from a target video dataset by using link relationships between Web videos obtained via metadata “related videos” and heterogeneous video features. Furthermore, by using the representative Web videos, we construct a network whose nodes and edges respectively correspond to Web videos and links between these Web videos. Then Web video groups, i.e., Web video sets with similar topics are hierarchically extracted based on strongly connected components, edge betweenness and modularity. By exhibiting the obtained hierarchical structure of Web video groups, users can easily grasp the overview of many Web videos. Consequently, even if users cannot write suitable queries that identify the desired contents, it becomes feasible to accurately retrieve the desired Web videos by selecting Web video groups according to the hierarchical structure. Experimental results on actual Web videos verify the effectiveness of our method.     

Towards 6DoF live video streaming system for immersive media

Based on the three rotational degrees (video in three dimensions, on the X, Y and Z axes) of freedom provided by VR, the viewer is free to control the viewing point and has six degrees of freedom (6DoF). When watching a sports game, the audience is no longer limited by the position of the camera, and can freely choose the viewing angle and position just like watching in the real world, which can greatly improve the immersion of viewing. However, the major barrier that prevents 6DoF video live from being industrialized lies in the extremely high computational complexity, of which multi-view depth estimation and Depth Image Based Rendering (DIBR) is difficult to realize. And existing devices do not have hardware interfaces that support multi-views coding technology. Therefore, we need new technologies for depth estimation and virtual view synthesis, and we need to use existing hardware coding/decoding interfaces to reduce power consumption. In this paper, we provide a 6DoF live video system, which includes multi-view depth estimation technique based on unsupervised learning, virtual viewpoint real-time rendering technology and 6DoF video coding. Experimental results demonstrate that our proposed acceleration method can speed up the original depth estimation algorithm by more than 34x, and can speed up the original DIBR algorithm by more than 168x. With our 6DoF video coding method, experimental results show that the bit rate achieves an average of 70%, 64%, 33%, 60% and 66% bitrate saving for AVC, HEVC, AV1, AVS3, VVC codec standard respectively.

     

一种基于用户播放行为序列的个性化视频推荐策略

本文针对在线视频服务网站的个性化推荐问题,提出了一种基于用户播放行为序列的个性化推荐策略.该策略通过深度神经网络词向量模型分析用户播放视频行为数据,将视频映射成等维度的特征向量,提取视频的语义特征.聚类用户播放历史视频的特征向量,建模用户兴趣分布矩阵.结合用户兴趣偏好和用户观看历史序列生成推荐列表.在大规模的视频服务系统中进行了离线实验,相比随机算法、基于物品的协同过滤和基于用户的协同过滤传统推荐策略,本方法在用户观看视频的Top-N推荐精确率方面平均分别获得22.3%、30.7%和934%的相对提升,在召回率指标上分别获得52.8%、41%和1065%的相对提升.进一步地与矩阵分解算法SVD++、基于双向LSTM模型和注意力机制的Bi-LSTM+Attention算法和基于用户行为序列的深度兴趣网络DIN比较,Top-N推荐精确率和召回率也得到了明显提升.该推荐策略不仅获得了较高的精确率和召回率,还尝试解决传统推荐面临大规模工业数据集时的数据要求严苛、数据稀疏和数据噪声等问题.     

CPA-VoD: Cloud and Peer-Assisted Video on Demand System for Mobile Devices

With the rapid development of WiFi and 3G/4G, people tend to view videos on mobile devices. These devices are ubiquitous but have small memory to cache videos. As a result, in contrast to traditional computers, these devices aggravate the network pressure of content providers. Previous studies use CDN to solve this problem. But its static leasing mechanism in which the rental space cannot be dynamically adjusted makes the operational cost soar and incompatible with the dynamically video delivery. In our study, based on a thorough analysis of user behavior from Tencent Video, a popular Chinese on-line video share platform, we identify two key user behaviors. Firstly, lots of users in the same region tend to watch the same video. Secondly, the popularity distribution of videos conforms with the Pareto principle, i.e., the top 20% popular videos own 80% of all video traffic. To turn these observations into silver bullet, we propose and implement a novel cloud- and peer-assisted video on demand system (CPA-VoD). In the system, we group users in the same region as a peer swarm, and in the same peer swarm, users can provide videos to other users by sharing their cached videos. Besides, we cache the 10% most popular videos in cloud servers to further alleviate the network pressure. We choose cloud servers to cache videos because the rental space can be dynamically adjusted. According to the evaluation on a real dataset from Tencent Video, CPA-VoD alleviates the network pressure and the operation cost excellently, while only 20.9% traffic is serviced by the content provider.     

一种Web视频监控播放插件

针对不同设备厂商的视频不能在统一界面集中监控的问题,设计了一种Web视频监控播放插件,该插件根据网络配置与前端视频设备直连通信,集成不同视频设备厂商提供的API,实现不同厂商设备的兼容. 基于ActiveX和NPAPI的插件实现机制,开发了实时预览、历史录像下载与回放和本地录像文件播放等功能接口,为不同浏览器的使用者提供相同的远程视频监控体验. 该插件在山东润兴化工智慧园区建设项目中得到了成功应用.     

Physically Based Video Editing

Convincing manipulation of objects in live action videos is a difficult and often tedious task. Skilled video editors achieve this with the help of modern professional tools, but complex motions might still lack physical realism since existing tools do not consider the laws of physics. On the other hand, physically based simulation promises a high degree of realism, but typically creates a virtual 3D scene animation rather than returning an edited version of an input live action video. We propose a framework that combines video editing and physics‐based simulation. Our tool assists unskilled users in editing an input image or video while respecting the laws of physics and also leveraging the image content. We first fit a physically based simulation that approximates the object's motion in the input video. We then allow the user to edit the physical parameters of the object, generating a new physical behavior for it. The core of our work is the formulation of an image‐aware constraint within physics simulations. This constraint manifests as external control forces to guide the object in a way that encourages proper texturing at every frame, yet producing physically plausible motions. We demonstrate the generality of our method on a variety of physical interactions: rigid motion, multi‐body collisions, clothes and elastic bodies.     

Affection arousal based highlight extraction for soccer video

Highlight extraction, as one of the key technologies in soccer video retrieval and summarization, has great academic and application value. According to the principle that the observer’s affection state would fluctuate with the evolution of game process when watching soccer match video, a novel highlight extraction approach based on the improved affection arousal model is proposed. Compared with the existing works, our main contributions include the following. A novel feature – shot intensity is exploited to replace the motion activity, which greatly improves the computational performance of affection arousal model. Another helpful feature – replay factor is designed and successfully fused into the affection arousal model. This makes the affection arousal model reflect the variation of the true match process more accurately. In addition, event temporal transition pattern (ETTP) in soccer video is utilized to detect highlights boundaries effectively combined with the affection arousal curve. Experiments conducted on real-world soccer game videos have demonstrated the efficiency and effectiveness of the proposed approach.     

Content-adaptive wireless streaming of instructional videos

Video streaming over wireless networks is becoming increasingly important for a variety of applications. To accommodate the dynamic change of wireless network bandwidths, Quality of Service (QoS) scalable video streams need to be provided. This paper presents a system of content-adaptive streaming of instructional (lecture) videos over wireless networks for E-learning applications. We first provide a real-time content analysis method to detect and extract content regions from instructional videos, then apply a “leaking-video-buffer” model to adjust QoS of video streams dynamically based on video content. In content-adaptive video streaming, an adaptive feedback control scheme is also developed to transmit properly compressed video streams to video clients not only based on network bandwidth, but also based on video content and the preferences of users. Finally, we demonstrate the scalability and content adaptiveness of the proposed video streaming system with experimental results on several instructional videos.     

Content-adaptive wireless streaming of instructional videos

Video streaming over wireless networks is becoming increasingly important for a variety of applications. To accommodate the dynamic change of wireless network bandwidths, Quality of Service (QoS) scalable video streams need to be provided. This paper presents a system of content-adaptive streaming of instructional (lecture) videos over wireless networks for E-learning applications. We first provide a real-time content analysis method to detect and extract content regions from instructional videos, then apply a “leaking-video-buffer” model to adjust QoS of video streams dynamically based on video content. In content-adaptive video streaming, an adaptive feedback control scheme is also developed to transmit properly compressed video streams to video clients not only based on network bandwidth, but also based on video content and the preferences of users. Finally, we demonstrate the scalability and content adaptiveness of the proposed video streaming system with experimental results on several instructional videos.     

A music-driven system for generating apparel display video

Video plays a great important role in online apparel sales, which is a vital tool for publicity and to provide consumers with space of imagination. However, as the apparel market rapidly updates in large amounts every day, creating videos for fast increasing clothes can be challenging and labor-consuming. Considering this, we present ApVideor, a music-driven video generation system customized for displaying clothes. This system consists of two main modules: music recommendation module and audio-visual synthesis module. The former assists users in searching background music that matches the apparel style, while the latter combines the audio and visuals into a video by music-driven approaches. Our user study suggests that this system makes the video creation process significantly easier and faster than manual creation. Meanwhile, the viewer test suggests that apparel-displaying videos created using our system are of comparable quality to those created manually by people who have worked with video editing.

     

Integrated approach of streaming 3d multimedia contents in real-time for mobile devices

It is popular to watch a 3D video through a 3D display nowadays. However, it is still difficult to enjoy the 3D multimedia contents with a mobile device even if a mobile device with a 3D display is currently introduced into the market. The main technological challenges for watching 3D contents via the mobile devices can be identified as the following: generating and streaming 3D contents. Generating 3D contents requires extra computational resources. Moreover, streaming 3D contents demands additional network bandwidth for receiving and transmitting the 3D data. To overcome these technological challenges, we propose ReMA, a novel 3D video streaming system in this paper. We devised a novel architecture for transmitter, receiver, and a distribution system to efficiently disseminate and generate 3D videos for the mobile devices. We implemented ReMA in a real test-bed and conducted a thorough empirical evaluation study to see the feasibility of streaming 3D contents for the mobile devices. Based on our empirical study, the resulting system presents a great promise in streaming 3D video in real-time to the mobile devices.     

Personal driving diary: Automated recognition of driving events from first-person videos

In this paper, we introduce the concept of personal driving diary. A personal driving diary is a multimedia archive of a person’s daily driving experience, describing important driving events of the user with annotated videos. This paper presents an automated system that constructs such multimedia diary by analyzing videos obtained from a vehicle-mounted camera. The proposed system recognizes important interactions between the driving vehicle and the other actors in videos (e.g., accident, overtaking, etc.), and labels them together with its contextual knowledge on the vehicle (e.g., mean velocity) to construct an event log. A decision tree based activity recognizer is designed, detecting driving events of vehicles and pedestrians from the first-person view videos by analyzing their trajectories and spatio-temporal relationships. The constructed diary enables efficient searching and event-based browsing of video clips, which helps the users when retrieving videos of dangerous situations. Our experiment confirms that the proposed system reliably generates driving diaries by annotating the vehicle events learned from training examples.