Towards effective video annotation: An approach to automatically link notes with video content

The characteristics of annotations, such as highlighting, context-based notes, and organization are difficult to translate from the traditional paper-based medium to the digital format. An added challenge is how to facilitate annotations on a digital video in a collaborative distance learning environment. To explore issues in video annotation, we developed a tool called Interactive Shared Education Environment (ISEE). ISEE automatically generates hyperlinked timestamps, which we called Smartlinks, to associate the notes with their video contents. A usability study with 59 participants, following up by a small-scale eye-tracking study, was conducted to explore users’ video note-taking behaviors and to examine the effect of the new Smartlink design. Our results showed that participants with Smartlink took fewer notes, focused less on video controls and more on video content than those without Smartlink. We believe the main benefit of Smartlink is that it may offload non-learning related cognitive loads and allow users to take better notes. Findings from this study on users’ video annotation behaviors shed light on the future design of video annotation systems in both individual and collaborative environments.     

Exploring Social Annotations with the Application to Web Page Recommendation

Collaborative social annotation systems allow users to record and share their original keywords or tag attachments to Web resources such as Web pages, photos, or videos. These annotations are a method for organizing and labeling information. They have the potential to help users navigate the Web and locate the needed resources. However, since annotations are posted by users under no central control, there exist problems such as spam and synonymous annotations. To efficiently use annotation information to facilitate knowledge discovery from the Web, it is advantageous if we organize social annotations from semantic perspective and embed them into algorithms for knowledge discovery. This inspires the Web page recommendation with annotations, in which users and Web pages are clustered so that semantically similar items can be related. In this paper we propose four graphic models which cluster users, Web pages and annotations and recommend Web pages for given users by assigning items to the right cluster first. The algorithms are then compared to the classical collaborative filtering recommendation method on a real-world data set. Our result indicates that the graphic models provide better recommendation performance and are robust to fit for the real applications.     

Open annotations on multimedia Web resources

Many Web portals allow users to associate additional information with existing multimedia resources such as images, audio, and video. However, these portals are usually closed systems and user-generated annotations are almost always kept locked up and remain inaccessible to the Web of Data. We believe that an important step to take is the integration of multimedia annotations and the Linked Data principles. We present the current state of the Open Annotation Model, explain our design rationale, and describe how the model can represent user annotations on multimedia Web resources. Applying this model in Web portals and devices, which support user annotations, should allow clients to easily publish and consume, thus exchange annotations on multimedia Web resources via common Web standards.     

针对用户兴趣的视频精彩片段提取

目的 视频精彩片段提取是视频内容标注、基于内容的视频检索等领域的热点研究问题。视频精彩片段提取主要根据视频底层特征进行精彩片段的提取,忽略了用户兴趣对于提取结果的影响,导致提取结果可能与用户期望不相符。另一方面,基于用户兴趣的语义建模需要大量的标注视频训练样本才能获得较为鲁棒的语义分类器,而对于大量训练样本的标注费时费力。考虑到互联网中包含内容丰富且易于获取的图像,将互联网图像中的知识迁移到视频片段的语义模型中可以减少大量的视频数据标注工作。因此,提出利用互联网图像的用户兴趣的视频精彩片段提取框架。方法 利用大量互联网图像对用户兴趣语义进行建模,考虑到从互联网中获取的知识变化多样且有噪声,如果不加选择盲目地使用会影响视频片段提取效果,因此,将图像根据语义近似性进行分组,将语义相似但使用不同关键词检索得到的图像称为近义图像组。在此基础上,提出使用近义语义联合组权重模型权衡,根据图像组与视频的语义相关性为不同图像组分配不同的权重。首先,根据用户兴趣从互联网图像搜索引擎中检索与该兴趣语义相关的图像集,作为用户兴趣精彩片段提取的知识来源;然后,通过对近义语义图像组的联合组权重学习,将图像中习得的知识迁移到视频中;最后,使用图像集中习得的语义模型对待提取片段进行精彩片段提取。结果 本文使用CCV数据库中的视频对本文提出的方法进行验证,同时与多种已有的视频关键帧提取算法进行比较,实验结果显示本文算法的平均准确率达到46.54,较其他算法相比提高了21.6%,同时算法耗时并无增加。此外,为探究优化过程中不同平衡参数对最终结果的影响,进一步验证本文方法的有效性,本文在实验过程中通过移除算法中的正则项来验证每一项对于算法框架的影响。实验结果显示,在移除任何一项后算法的准确率明显降低,这表明本文方法所提出的联合组权重模型对提取用户感兴趣视频片段的有效性。结论 本文提出了一种针对用户兴趣语义的视频精彩片段提取方法,根据用户关注点的不同,为不同用户提取其感兴趣的视频片段。     

Videolization: knowledge graph based automated video generation from web content

Web content nowadays can also be accessed through new generation of Internet connected TVs. However, these products failed to change users’ behavior when consuming online content. Users still prefer personal computers to access Web content. Certainly, most of the online content is still designed to be accessed by personal computers or mobile devices. In order to overcome the usability problem of Web content consumption on TVs, this paper presents a knowledge graph based video generation system that automatically converts textual Web content into videos using semantic Web and computer graphics based technologies. As a use case, Wikipedia articles are automatically converted into videos. The effectiveness of the proposed system is validated empirically via opinion surveys. Fifty percent of survey users indicated that they found generated videos enjoyable and 42 % of them indicated that they would like to use our system to consume Web content on their TVs.

     

在线教育的建议与设计——“立体化”视频播放器

在线教育发展如火如荼,笔者通过分析当下在线教育平台,发现目前在线教育存在着不足。该文以C/S体系结构设计客户端和后台服务器,实现用户在观看在线教育视频中标记笔记,笔记共享和拓展知识的及时推送功能。通过该设计,减少当前在线教育平台“扁平化”的教育模式,防止用户“被动”观看视频,提高用户学习效率。     

Annotations and the collaborative digital library: Effects of an aligned annotation interface on student argumentation and reading strategies

Recent research on annotation interfaces provides provocative evidence that anchored, annotation-based discussion environments may lead to better conversations about a text. However, annotation interfaces raise complicated tradeoffs regarding screen real estate and positioning. It is argued that solving this screen real estate problem requires limiting the number of annotations displayed to users. In order to understand which annotations have the most learning value for students, this paper presents two complementary studies examining the effects of annotations on students performing a reading-to-write task. The first study used think-aloud protocols and a within-subjects methodology, finding that annotations appeared to provoke students to reflect more critically upon the primary text. This effect was particularly strong when students encountered pairs of annotations presenting different viewpoints on the same section of text. Student interviews suggested that annotations were most helpful when they caused the reader to consider and weigh conflicting viewpoints. The second study used a between-subjects methodology and a more naturalistic task to provide complementary evidence that annotations encourage more reflective responses to a text. This study found that students who received annotated materials both perceived themselves and were perceived by instructors as less reliant on unreflective summary strategies than students who received the same content but in a different format. These findings indicate that the learning value of an annotation lies in its ability to provoke students to consider and weigh new perspectives on the primary text. When selected effectively, annotations provide a critical scaffolding that can support students’ critical thinking and argumentation activities. Collaborative digital libraries and applications for the Web 2.0 should be designed with this learning framework in mind.     

A Web video retrieval method using hierarchical structure of Web video groups

In this paper, we propose a Web video retrieval method that uses hierarchical structure of Web video groups. Existing retrieval systems require users to input suitable queries that identify the desired contents in order to accurately retrieve Web videos; however, the proposed method enables retrieval of the desired Web videos even if users cannot input the suitable queries. Specifically, we first select representative Web videos from a target video dataset by using link relationships between Web videos obtained via metadata “related videos” and heterogeneous video features. Furthermore, by using the representative Web videos, we construct a network whose nodes and edges respectively correspond to Web videos and links between these Web videos. Then Web video groups, i.e., Web video sets with similar topics are hierarchically extracted based on strongly connected components, edge betweenness and modularity. By exhibiting the obtained hierarchical structure of Web video groups, users can easily grasp the overview of many Web videos. Consequently, even if users cannot write suitable queries that identify the desired contents, it becomes feasible to accurately retrieve the desired Web videos by selecting Web video groups according to the hierarchical structure. Experimental results on actual Web videos verify the effectiveness of our method.     

Webvise: browser and proxy support for open hypermedia structuring mechanisms on the World Wide Web

This paper discusses how to augment the World Wide Web with an open hypermedia service (Webvise) that provides structures such as contexts, links, annotations, and guided tours stored in hypermedia databases external to the Web pages. This includes the ability for users collaboratively to create links from parts of HTML Web pages they do not own and support for creating links to parts of Web pages without writing HTML target tags. The method for locating parts of Web pages can locate parts of pages across frame hierarchies and it also supports certain repairs of links that break due to modified Web pages. Support for providing links to/from parts of non-HTML data, such as sound and movie, will be possible via interfaces to plug-ins and Java-based media players.The hypermedia structures are stored in a hypermedia database, developed from the Devise Hypermedia framework, and the service is available on the Web via an ordinary URL. The best user interface for creating and manipulating the structures is currently provided for the Microsoft Internet Explorer 4.x browser through COM integration that utilizes the Explorer's DOM representation of Web-pages. But the structures can also be manipulated and used via special Java applets and a pure proxy server solution is provided for users who only need to browse the structures. A user can create and use the external structures as `transparency' layers on top of arbitrary Web pages, the user can switch between viewing pages with one or more layers (contexts) of structures or without any external structures imposed on them.     

ShutterApp: Spatio‐temporal Exposure Control for Videos

A camera's shutter controls the incoming light that is reaching the camera sensor. Different shutters lead to wildly different results, and are often used as a tool in movies for artistic purpose, e.g., they can indirectly control the effect of motion blur. However, a physical camera is limited to a single shutter setting at any given moment. ShutterApp enables users to define spatio‐temporally‐varying virtual shutters that go beyond the options available in real‐world camera systems. A user provides a sparse set of annotations that define shutter functions at selected locations in key frames. From this input, our solution defines shutter functions for each pixel of the video sequence using a suitable interpolation technique, which are then employed to derive the output video. Our solution performs in real‐time on commodity hardware. Hereby, users can explore different options interactively, leading to a new level of expressiveness without having to rely on specialized hardware or laborious editing.     

一种基于社会性标注的网页排序算法

社会性标注作为一种新的资源管理和共享方式,吸引为数众多的用户参与其中,由此产生的大量社会性标注数据成为网页质量评价的一个新维度.文中研究如何利用社会性标注改进网页检索性能,提出一种有机结合网页和用户的查询相关性与互增强关系的网页排序算法.首先利用统计主题模型,使用相关标签为网页和用户建模,并计算查询相关性.然后利用二部图模型刻画网页和用户间的互增强关系,并使用相关标签与用户兴趣和网页内容的匹配度为互增强关系赋予权重.最后结合查询相关性和互增强关系,以迭代方式同时计算网页和用户的评分.实验结果表明,文中提出的检索模型和互增强模型能够有效地提高排序算法的性能.与目前的代表性算法相比,该算法在检索性能上有明显提高.     

基于Ajax和FMS的视频直播学习系统设计与实现

介绍了一种基于Ajax和FMS的视频直播学习系统。该系统采用B/S结构,应用支持异步请求的Ajax技术提供即时通信,利用基于RTMP协议的FMS提供流媒体服务。由于客户端只需要一个支持Flash的网页浏览器,并且具有占用带宽少、响应及时、视频质量高和容易部署等特点,该系统在远程教育等领域具有较大的应用前景。     

基于Web的网络教学平台的设计

基于Web的网络教学平台是开展网络化学习的基础环境,它为教师、学生和教学管理人员提供一个学习和工作的网络环境。本文介绍了一个包括文章管理、课件下载、视频点播、教学论坛、离线作业、网上考试、师生交互的多媒体网络教学平台的设计,该平台采用层次化设计结构,通过提供一个个性化的门户和一系列的支持工具,将相互独立的教学服务子系统集成起来,为用户提供统一的、个性化的教学服务。全面介绍了平台的设计原则、用户分类和系统功能,并着重介绍系统结构。     

How Software Developers Use Tagging to Support Reminding and Refinding

Developers frequently add annotations to source code to help them remember pertinent information and mark locations of interest for future investigation. Finding and refinding these notes is a form of navigation that is integral to software maintenance. Although there is some tool support in modern development environments for authoring and navigating these comments, we have observed that these annotations often fail to remind and are sometimes difficult to find by the programmer. To address these shortcomings, we have designed a new approach for software navigation called Tags for Software Engineering Activities (TagSEA). TagSEA combines the notion of waypointing (a mechanism for marking locations in spatial navigation) with social tagging to support programmers in defining semantically rich annotations to source code comments. The tool provides support for creating, editing, navigating, and managing these annotations. We present the results from two empirical studies, where we observed and then analyzed how professional programmers used source code annotations to support their development activities over 24 months. Our findings indicate that the addition of semantic information to annotations can improve their value. We also provide suggestions on how annotation tools in general may be improved.     

流式媒体同步集成技术在远程教学信息系统中的应用

现代教育技术并不等于现代远程教育,在教学过程中使用了现代信息技术并不就是现代远程教育。如何将数字视频／音频信息融于Web授课课件,这是当前远程教学和教育技术领域的新课题,该文介绍远程教学信息系统建设过程中进行的探索和实践。     

轻量化Transformer目标跟踪数据标注算法

目的 基于深度模型的跟踪算法往往需要大规模的高质量标注训练数据集,而人工逐帧标注视频数据会耗费大量的人力及时间成本。本文提出一个基于Transformer模型的轻量化视频标注算法（Transformer-based label network,TLNet）,实现对大规模稀疏标注视频数据集的高效逐帧标注。方法 该算法通过Transformer模型来处理时序的目标外观和运动信息,并融合前反向的跟踪结果。其中质量评估子网络用于筛选跟踪失败帧,进行人工标注;回归子网络则对剩余帧的初始标注进行优化,输出更精确的目标框标注。该算法具有强泛化性,能够与具体跟踪算法解耦,应用现有的任意轻量化跟踪算法,实现高效的视频自动标注。结果 在2个大规模跟踪数据集上生成标注。对于LaSOT （large-scale single object tracking）数据集,自动标注过程仅需约43 h,与真实标注的平均重叠率（mean intersection over union,mIoU）由0.824提升至0.871。对于TrackingNet数据集,本文使用自动标注重新训练3种跟踪算法,并在3个数据集上测试跟踪性能,使用本文标注训练的模型在跟踪性能上超过使用TrackingNet原始标注训练的模型。结论 本文算法TLNet能够挖掘时序的目标外观和运动信息,对前反向跟踪结果进行帧级的质量评估并进一步优化目标框。该方法与具体跟踪算法解耦,具有强泛化性,并能节省超过90%的人工标注成本,高效地生成高质量的视频标注。     

STIMO: STIll and MOving video storyboard for the web scenario

In the current Web scenario a video browsing tool that produces on-the-fly storyboards is more and more a need. Video summary techniques can be helpful but, due to their long processing time, they are usually unsuitable for on-the-fly usage. Therefore, it is common to produce storyboards in advance, penalizing users customization. The lack of customization is more and more critical, as users have different demands and might access the Web with several different networking and device technologies. In this paper we propose STIMO, a summarization technique designed to produce on-the-fly video storyboards. STIMO produces still and moving storyboards and allows advanced users customization (e.g., users can select the storyboard length and the maximum time they are willing to wait to get the storyboard). STIMO is based on a fast clustering algorithm that selects the most representative video contents using HSV frame color distribution. Experimental results show that STIMO produces storyboards with good quality and in a time that makes on-the-fly usage possible.     

弹幕信息协助下的视频多标签分类

文中探究了弹幕信息协助下的视频多标签分类任务。多标签视频分类任务根据视频内容从不同角度赋予视频多个标签,与视频推荐等应用紧密相关。多标签视频数据集的高标注成本和对视频内容的多角度理解是该研究领域面临的主要问题。弹幕是一种新近出现的用户评论形式,受到了众多用户的欢迎。由于用户参与度高,弹幕视频网站的视频拥有大量用户自发添加的标签,这些标签是天然的多标签数据。文中以此构建了一个多标签视频数据集,并整理出了视频标签间的层级语义关系,该数据集在未来将公开发布。同时,弹幕文本模态包含大量与视频内容相关的细粒度信息,因此在以往视频分类工作融合视觉和音频模态的基础上,引入弹幕文本模态进行视频多标签分类研究。在基于聚类的NeXtVLAD模型、注意力Dbof模型和基于时序的GRU模型上进行实验,在增加弹幕模态后,GAP指标最高提升了23%,证明了弹幕信息对该任务具有辅助作用。此外,还探索了如何在分类中利用标签层级关系,通过构建标签关系矩阵来改造标签,进而将标签语义融入训练。实验结果表明,加入标签关系后,Hit@1指标提升了15%,因此其能优化多标签分类的效果。此外,MAP指标在细粒度小类上提升了4%,说明标签语义的引入有利于预测样本量较少的类别,具有研究价值。