Real-time panoramic video streaming system with overlaid interface concept for social media

In this paper, we propose a real-time panoramic video streaming system with overlaid interface concept for social media. The proposed system provides real-time panorama images for smart displays such as smart TVs, smart phones, and tablet PCs. Panorama images are collected at sporting events via panorama cameras. Contents thus collected are sent to servers and then provided to smart devices via live sports video streams. Users select a panorama camera and choose their viewing angle and zooming factor for the selected panorama camera. The proposed system provides immersive and realistic views of live sporting events on users’ displays. Furthermore, an overlaid panoramic interface concept is proposed for immersive live baseball watching combined with tightly integrated social media experience.     

一种基于用户播放行为序列的个性化视频推荐策略

本文针对在线视频服务网站的个性化推荐问题,提出了一种基于用户播放行为序列的个性化推荐策略.该策略通过深度神经网络词向量模型分析用户播放视频行为数据,将视频映射成等维度的特征向量,提取视频的语义特征.聚类用户播放历史视频的特征向量,建模用户兴趣分布矩阵.结合用户兴趣偏好和用户观看历史序列生成推荐列表.在大规模的视频服务系统中进行了离线实验,相比随机算法、基于物品的协同过滤和基于用户的协同过滤传统推荐策略,本方法在用户观看视频的Top-N推荐精确率方面平均分别获得22.3%、30.7%和934%的相对提升,在召回率指标上分别获得52.8%、41%和1065%的相对提升.进一步地与矩阵分解算法SVD++、基于双向LSTM模型和注意力机制的Bi-LSTM+Attention算法和基于用户行为序列的深度兴趣网络DIN比较,Top-N推荐精确率和召回率也得到了明显提升.该推荐策略不仅获得了较高的精确率和召回率,还尝试解决传统推荐面临大规模工业数据集时的数据要求严苛、数据稀疏和数据噪声等问题.     

Energy optimization for mobile video streaming via an aggregate model

Wireless video streaming on smartphones drains a significantly large fraction of battery energy, which is primarily consumed by wireless network interfaces for downloading unused data and repeatedly switching radio interface. In this paper, we propose an energy-efficient download scheduling algorithm for video streaming based on an aggregate model that utilizes user’s video viewing history to predict user behavior when watching a new video, thereby minimizing wasted energy when streaming over wireless network interfaces. The aggregate model is constructed by a personal retention model with users’ personal viewing history and the audience retention on crowd-sourced viewing history, which can accurately predict the user behavior of watching videos by balancing “user interest” and “video attractiveness”. We evaluate different users streaming multiple videos in various wireless environments and the results illustrate that the aggregate model can help reduce energy waste by 20 % on average. In addition, we also discuss implementation details and extensions, such as dynamically updating personal retention, balancing audience and personal retention, categorizing videos for accurate model.     

TabuVis: A tool for visual analytics multidimensional datasets

Efficient video summarization leads to facilely exploring video content appropriate to the user’s intention with low cognitive demand. In this paper, we present a novel approach for summarizing videos in the form of multi-scale structures that exhibit different video features at different scale levels and allow exploration of video contents with multi-scale interaction. The semantic relationship between structures is addressed and user intention is also considered and integrated in the summarization and interaction. This paper first introduces the concept of multi-scale structures for summarizing video content and describes three different types of structures that present important features at different scale levels. Furthermore, a continuous zooming interaction for browsing multi-scale structures is provided to facilitate video browsing. Finally, an elaborate user study is conducted showing that user performance on understanding and browsing videos is improved.     

A target advertisement system based on TV viewer’s profile reasoning

The traditional broadcasting services such as terrestrial, satellite and cable broadcasting have been unidirectional mass media regardless of TV viewer’s preferences. Recently rich media streaming has become possible via the broadband networks. Furthermore, since bidirectional communication is possible, personalcasting such as personalized streaming service has been emerging by taking into account the user’s preference on content genres, viewing times and actors/actresses etc. Accordingly personal media becomes an important means for content provision service in addition to the traditional broadcasting service as mass media. In this paper, we introduce a user profile reasoning method for TV viewers. The user profile reasoning is made in terms of genre preference and TV viewing times for TV viewer’s groups in different genders and ages. For user profiling reasoning, the TV viewing history data is used to train the proposed user profiling reasoning algorithm which allows for target advertisement for different age/gender groups. To show the effectiveness of our proposed user profile reasoning method, we present plenty of the experimental results by using real TV usage history.     

Multiscreen Social TV System: A Mixed Method Understanding of Users’ Attitudes and Adoption Intention

Past social TV (STV) system user studies usually tested a small number of trialists in lab settings in Western countries. This Asian research took a mix method approach (interview, observation, and web survey) to examine the attitudes of 76 young adults’ experiences of using a multiscreen STV system. The results show that the majority of participants preferred shifting and complementary patterns when using STV. Their favorite feature was multiscreen video teleportation via QR codes. Nearly half of participants agreed that sociability features of the STV system could facilitate their perceived social presence and sense of community, which increased adoption intention. Additionally, text messaging was selected as the preferred mode for social interactions during video viewing. Although most participants liked the user-friendly design to upload and share videos conveniently, age has a negative association with users’ receptiveness of the user generated video channel creation and aggregation feature.     

Transmission of correct gaze direction in video conferencing using screen-embedded cameras

In this paper, we propose a new video conferencing system that presents correct gaze directions of a remote user by switching among images obtained from multiple cameras embedded in a screen according to a local user’s position. Our proposed method reproduces a situation like that in which the remote user is in the same space as the local user. The position of the remote user to be displayed on the screen is determined so that the positional relationship between the users is reproduced. The system selects one of the embedded cameras whose viewing direction towards the remote user is the closest to the local user’s viewing direction to the remote user’s image on the screen. As a result of quantitative evaluation, we confirmed that, in comparison with the case using a single camera, the accuracy of gaze estimation was improved by switching among the cameras according to the position of the local user.

     

Seamless video access for mobile devices by content-aware utility-based adaptation

Today’s Internet multimedia services are characterized by heterogeneous networks, a wide range of terminals, diverse user preferences, and varying natural environment conditions. Heterogeneity of terminals, networks, and user preferences impose nontrivial challenges to the Internet multimedia services for providing seamless multimedia access particularly for mobile devices (e.g., laptops, tablet PCs, PDAs, mobile phones, etc.). Thus, it is essential that advanced multimedia technologies are developed to deal with these challenges. One of these technologies is video adaptation, which has gained significant importance with its main objective of enabling seamless access to video contents available over the Internet. Adaptation decision taking, which can be considered as the “brain” of video adaptation, assists video adaptation to achieve this objective. Scalable Video Coding (SVC) offers flexibility for video adaptation through providing a comprehensive set of scalability parameters (i.e., temporal, spatial, and quality) for producing scalable video streams. Deciding the best combination of scalability parameters to adapt a scalable video stream while satisfying a set of constraints (e.g., device specifics, network bandwidth, etc.) poses challenges for the existing adaptation services to enable seamless video access. To ease such challenges, an adaptation decision taking technique employing a utility-based approach to decide on the most adequate scalability parameters for adaptation operations is developed. A Utility Function (UF), which models the relationships among the scalability parameters and weights specifying the relative importance of these parameters considering video content characteristics (i.e., motion activity and structural feature), is proposed to assist the developed technique. In order to perform the developed adaptation decision taking technique, a video adaptation framework is also proposed in this paper. The adaptation experiments performed using the proposed framework prove the effectiveness of the framework to provide an important step towards enabling seamless video access for mobile devices to enhance viewing experience of users.     

一种视频质量监测系统的设计与实现

网络服务提供商希望能从用户的角度了解目前网络所提供的服务质量,而用户也希望获得定量的指标来评价当前网络服务质量。为此,以视频质量监测为研究对象,提出一种面向用户体验质量的网络监测系统。通过实验分析了网络传输过程中QoS参数对视频QoE的影响;提出一种将视频流转化为测试序列的视频丢包测量方法,该方法能低入侵、准确测量视频传输过程中的丢包情况;基于以上的研究成果,通过对MIB库的扩展和对MIB库轮询机制的研究,构建了面向QoE的视频服务监测系统,该监测系统结构简单、可行性强,实验表明可实时对网络中的视频服务质量进行监测。     

Search personalization through query and page topical analysis

Thousands of users issue keyword queries to the Web search engines to find information on a number of topics. Since the users may have diverse backgrounds and may have different expectations for a given query, some search engines try to personalize their results to better match the overall interests of an individual user. This task involves two great challenges. First the search engines need to be able to effectively identify the user interests and build a profile for every individual user. Second, once such a profile is available, the search engines need to rank the results in a way that matches the interests of a given user. In this article, we present our work towards a personalized Web search engine and we discuss how we addressed each of these challenges. Since users are typically not willing to provide information on their personal preferences, for the first challenge, we attempt to determine such preferences by examining the click history of each user. In particular, we leverage a topical ontology for estimating a user’s topic preferences based on her past searches, i.e. previously issued queries and pages visited for those queries. We then explore the semantic similarity between the user’s current query and the query-matching pages, in order to identify the user’s current topic preference. For the second challenge, we have developed a ranking function that uses the learned past and current topic preferences in order to rank the search results to better match the preferences of a given user. Our experimental evaluation on the Google query-stream of human subjects over a period of 1 month shows that user preferences can be learned accurately through the use of our topical ontology and that our ranking function which takes into account the learned user preferences yields significant improvements in the quality of the search results.     

A cognitive approach for agent-based personalized recommendation

There is an increasing need for various e-service, e-commerce and e-business sites to provide personalized recommendations to on-line customers. This paper proposes a new type of personalized recommendation agents called fuzzy cognitive agents. Fuzzy cognitive agents are designed to give personalized suggestions based on the user’s current personal preferences, other user’s common preferences, and expert’s domain knowledge. Fuzzy cognitive agents are able to represent knowledge via extended fuzzy cognitive maps, to learn users’ preferences from most recent cases and to help customers make inferences and decisions through numeric computation instead of symbolic and logic deduction. A case study is included to illustrate how personalized recommendations are made by fuzzy cognitive agents in e-commerce sites. The case study demonstrates that the fuzzy cognitive agent is both flexible and effective in supporting e-commerce applications.     

An innovative web-based collaborative platform for video annotation

Large scale labeled datasets are of key importance for the development of automatic video analysis tools as they, from one hand, allow multi-class classifiers training and, from the other hand, support the algorithms’ evaluation phase. This is widely recognized by the multimedia and computer vision communities, as witnessed by the growing number of available datasets; however, the research still lacks in annotation tools able to meet user needs, since a lot of human concentration is necessary to generate high quality ground truth data. Nevertheless, it is not feasible to collect large video ground truths, covering as much scenarios and object categories as possible, by exploiting only the effort of isolated research groups. In this paper we present a collaborative web-based platform for video ground truth annotation. It features an easy and intuitive user interface that allows plain video annotation and instant sharing/integration of the generated ground truths, in order to not only alleviate a large part of the effort and time needed, but also to increase the quality of the generated annotations. The tool has been on-line in the last four months and, at the current date, we have collected about 70,000 annotations. A comparative performance evaluation has also shown that our system outperforms existing state of the art methods in terms of annotation time, annotation quality and system’s usability.     

Joint In-network Video Rate Adaptation and Measurement-Based Admission Control: Algorithm Design and Evaluation

The important new revenue opportunities that multimedia services offer to network and service providers come with important management challenges. For providers, it is important to control the video quality that is offered and perceived by the user, typically known as the quality of experience (QoE). Both admission control and scalable video coding techniques can control the QoE by blocking connections or adapting the video rate but influence each other’s performance. In this article, we propose an in-network video rate adaptation mechanism that enables a provider to define a policy on how the video rate adaptation should be performed to maximize the provider’s objective (e.g., a maximization of revenue or QoE). We discuss the need for a close interaction of the video rate adaptation algorithm with a measurement based admission control system, allowing to effectively orchestrate both algorithms and timely switch from video rate adaptation to the blocking of connections. We propose two different rate adaptation decision algorithms that calculate which videos need to be adapted: an optimal one in terms of the provider’s policy and a heuristic based on the utility of each connection. Through an extensive performance evaluation, we show the impact of both algorithms on the rate adaptation, network utilisation and the stability of the video rate adaptation. We show that both algorithms outperform other configurations with at least 10 %. Moreover, we show that the proposed heuristic is about 500 times faster than the optimal algorithm and experiences only a performance drop of approximately 2 %, given the investigated video delivery scenario.     

基于云存储的3D直播系统设计与实现

随着互联网技术的快速发展,观看网络直播成为人们主要的线上娱乐方式之一。目前市面上存在很多直播平台,在这些平台日趋广泛的同时,也产生了很多改进的空间,如何增强用户的临场感体验是其中至关重要的一个方面。一般在实际观看中,用户所看到的是2D画面,观看体验感不强,且主播画面与用户实时显示画面,时延大概在2s以上,互动感较差。采用3D直播的方式,将主播以三维虚拟人物形象的方式呈现,将增强用户的体验感。而现有的3D直播平台存在一定的局限性,用户依然是被动的观看直播画面,因此对现有直播平台架构进行了研究和改进。基于Unity软件,采用C/S架构,将人物渲染的流程从局域网拓展到广域网中,在客户端将被动观看变为主动观看,再通过引入CDN和5G网络技术降低传输时延,完成3D直播系统平台设计与搭建。通过该平台可拉近用户与主播的心理距离,增加用户观看网络直播的趣味性与临场感体验。     

Modeling user perception of 3D video based on ambient illumination context for enhanced user centric media access and consumption

For enjoying 3D video to its full extent, it is imperative that access and consumption of it is user centric, which in turn ensures improved 3D video perception. Several important factors including video characteristics, users’ preferences, contexts prevailing in various usage environments, etc have influences on 3D video perception. Thus, to assist efficient provision of user centric media, user perception of 3D video should be modeled considering the factors affecting perception. Considering ambient illumination context to model 3D video perception is an interesting research topic, which has not been particularly investigated in literature. This context is taken into account while modeling video quality and depth perception of 3D video in this paper. For the video quality perception model: motion and structural feature characteristics of color texture sequences; and for the depth perception model: luminance contrast of color texture and depth intensity of depth map sequences of 3D video are used as primary content related factors in the paper. Results derived using the video quality and depth perception models demonstrate that these models can efficiently predict user perception of 3D video considering the ambient illumination context in user centric media access and consumption environments.     

Learning for automatic personalization in a semantic taxonomy-based meta-search agent

Providing highly relevant page hits to the user is a major concern in Web search. To accomplish this goal, the user must be allowed to express his intent precisely. Secondly, page hit rating mechanisms should be used that take the user’s intent into account. Finally, a learning mechanism is needed that captures a user’s preferences in his Web search, even when those preferences are changing dynamically. To address the first two issues, we propose a semantic taxonomy-based meta-search agent approach that incorporates the user’s taxonomic search intent. It also addresses relevancy improvement issues of the resulting page hits by using user’s search intent and preference-based rating. To provide a learning mechanism, we first propose a connectionist model-based user profile representation approach, which can leverage all of the features of the semantic taxonomy-based information retrieval approach. A user profile learning algorithm is also devised for our proposed user profile representation framework by significantly modifying and extending a typical neural network learning algorithm. Finally, the entire methodology including this learning mechanism is implemented in an agent-based system, WebSifter II. Empirical results of learning performance are also discussed.     

A DASH-based HEVC multi-view video streaming system

Recent advancement in cameras and image processing technology has generated a paradigm shift from traditional 2D and 3D video to multi-view video (MVV) technology, while at the same time improving video quality and compression through standards such as high efficiency video coding (HEVC). In multi-view, cameras are placed in predetermined positions to capture the video from various views. Delivering such views with high quality over the Internet is a challenging prospect, as MVV traffic is several times larger than traditional video, since it consists of multiple video sequences, each captured from a different angle, requiring more bandwidth than single-view video to transmit MVV. In addition, the Internet is known to be prone to packet loss, delay, and bandwidth variation, which adversely affect MVV transmission. Another challenge is that end users’ devices have different capabilities in terms of computing power, display, and access link capacity, requiring MVV to be adapted to each user’s context. In this paper, we propose an HEVC multi-view system using Dynamic Adaptive Streaming over HTTP to overcome the above-mentioned challenges. Our system uses an adaptive mechanism to adjust the video bit rate to the variations of bandwidth in best effort networks. We also propose a novel scalable way for the multi-view video and depth content for 3D video in terms of the number of transmitted views. Our objective measurements show that our method of transmitting MVV content can maximize the perceptual quality of virtual views after the rendering and hence increase the user’s quality of experience.     

Racking focus and tracking focus on live video streams: a stereo solution

The ability to produce dynamic Depth of Field effects in live video streams was until recently a quality unique to movie cameras. In this paper, we present a computational camera solution coupled with real-time GPU processing to produce runtime dynamic Depth of Field effects. We first construct a hybrid-resolution stereo camera with a high-res/low-res camera pair. We recover a low-res disparity map of the scene using GPU-based Belief Propagation, and subsequently upsample it via fast Cross/Joint Bilateral Upsampling. With the recovered high-resolution disparity map, we warp the high-resolution video stream to nearby viewpoints to synthesize a light field toward the scene. We exploit parallel processing and atomic operations on the GPU to resolve visibility when multiple pixels warp to the same image location. Finally, we generate racking focus and tracking focus effects from the synthesized light field rendering. All processing stages are mapped onto NVIDIA’s CUDA architecture. Our system can produce racking and tracking focus effects for the resolution of 640×480 at 15 fps.     

An HTTP/2 Push-Based Approach for Low-Latency Live Streaming with Super-Short Segments

Over the last years, streaming of multimedia content has become more prominent than ever. To meet increasing user requirements, the concept of HTTP Adaptive Streaming (HAS) has recently been introduced. In HAS, video content is temporally divided into multiple segments, each encoded at several quality levels. A rate adaptation heuristic selects the quality level for every segment, allowing the client to take into account the observed available bandwidth and the buffer filling level when deciding the most appropriate quality level for every new video segment. Despite the ability of HAS to deal with changing network conditions, a low average quality and a large camera-to-display delay are often observed in live streaming scenarios. In the meantime, the HTTP/2 protocol was standardized in February 2015, providing new features which target a reduction of the page loading time in web browsing. In this paper, we propose a novel push-based approach for HAS, in which HTTP/2’s push feature is used to actively push segments from server to client. Using this approach with video segments with a sub-second duration, referred to as super-short segments, it is possible to reduce the startup time and end-to-end delay in HAS live streaming. Evaluation of the proposed approach, through emulation of a multi-client scenario with highly variable bandwidth and latency, shows that the startup time can be reduced with 31.2% compared to traditional solutions over HTTP/1.1 in mobile, high-latency networks. Furthermore, the end-to-end delay in live streaming scenarios can be reduced with 4 s, while providing the content at similar video quality.     

Improving the Quality of the Personalized Electronic Program Guide

As Digital TV subscribers are offered more and more channels, it is becoming increasingly difficult for them to locate the right programme information at the right time. The personalized Electronic Programme Guide (pEPG) is one solution to this problem; it leverages artificial intelligence and user profiling techniques to learn about the viewing preferences of individual users in order to compile personalized viewing guides that fit their individual preferences. Very often the limited availability of profiling information is a key limiting factor in such personalized recommender systems. For example, it is well known that collaborative filtering approaches suffer significantly from the sparsity problem, which exists because the expected item-overlap between profiles is usually very low. In this article we address the sparsity problem in the Digital TV domain. We propose the use of data mining techniques as a way of supplementing meagre ratings-based profile knowledge with additional item-similarity knowledge that can be automatically discovered by mining user profiles. We argue that this new similarity knowledge can significantly enhance the performance of a recommender system in even the sparsest of profile spaces. Moreover, we provide an extensive evaluation of our approach using two large-scale, state-of-the-art online systems—PTVPlus, a personalized TV listings portal and Físchlár, an online digital video library system.