足球比赛场景的三维再现和自动解说

特定领域的视频解说巳成为当前多媒体研究的热点问题，其中一个主要应用领域是体育运动的解说与分析，文中介绍一个实用的足球比赛场景的三维重建和视频解说系统；首先通过足球比赛的视频序列恢复场景的三维模型，包括球员的位置、速度等参数，以及足球的运动轨迹，并根据这些信息和特定的先验知识识别球员的动作行为；进而对比赛过程加以评论；最后构造一个OpenGL的虚拟环境，用户可以在球场上漫游，从任意视角观看比赛。     

Live 3D Video in Soccer Stadium

This paper proposes a method to realize a 3D video system that can capture video data from multiple cameras, reconstruct 3D models, transmit 3D video streams via the network, and display them on remote PCs. All processes are done in real time. We represent a player with a simplified 3D model consisting of a single plane and a live video texture extracted from multiple cameras. This 3D model is simple enough to be transmitted via a network. A prototype system has been developed and tested at actual soccer stadiums. A 3D video of a typical soccer scene, which includes more than a dozen players, was processed at video rate and transmitted to remote PCs through the internet at 15–24 frames per second.     

The virtual museum: Interactive 3D navigation of a multimedia database

The Virtual Museum is an interactive, electronic museum where users can move from room to room, and select any exhibit in a room for more detailed examination. The exhibits in the museum are educational, encompassing topics such as medicine, plant growth, the environment, and space. To facilitate interaction with the museum, a new method for navigating through a prerendered 3D space, and interacting with objects in that space has been developed, called ‘virtual navigation’. Virtual navigation employs real-time video decompression for the display of, and interaction with, high-quality computer animation. In addition, a representation for 3D objects in animated sequences is used which permits pixel-accurate, frame-accurate object picking, so that a viewer can select any 3D object to trigger movement within the 3D space, to examine an exhibit in animated form, or to play a digital movie or soundtrack. The use of precomputed video permits 3D navigation in a realistic-looking space, without requiring special-purpose graphics hardware.     

Automatic initialization for 3D soccer player tracking

As a special application of computer vision, automatic sports video analysis has been studied by some researchers. This sports video analysis via computer vision is a moderately challenging problem: it is more difficult than analyzing a video of a few laboratory members acting as in a simple scenario and is easier than analyzing a video of crowded people at a subway station. So the success of an analysis heavily depends on how much one can exploit the prior information on the sport and setting. The most challenging and important part would be the tracking of players (and ball). With a multi-camera system, 3D tracking is feasible which is much more meaningful than 2D tracking for the analysis. As an initial step of 3D player tracking from multi-view soccer videos, this paper deals with automatic initialization of player positions. Initial 3D positions can be estimated by exploiting some conditions of a soccer match. To make it robust, prior knowledge on the features of players is learnt by support vector machines (SVM). Experimental results show that the proposed system is efficient for general soccer sequences.     

Modeling and rendering of realistic waterfall scenes with dynamic texture sprites

We propose an efficient approach for authoring dynamic and realistic waterfall scenes based on an acquired video sequence. Traditional video based techniques generate new images by synthesizing 2D samples, i.e., texture sprites chosen from a video sequence. However, they are limited to one fixed viewpoint and cannot provide arbitrary walkthrough into 3D scenes. Our approach extends this scheme by synthesizing dynamic 2D texture sprites and projecting them into 3D space. We first generate a set of basis texture sprites, which capture the representative appearance and motions of waterfall scenes contained in the video sequence. To model the shape and motion of a new waterfall scene, we interactively construct a set of flow lines taking account of physical principles. Along each flow line, the basis texture sprites are manipulated and animated dynamically, yielding a sequence of dynamic texture sprites in 3D space. These texture sprites are displayed using the point splatting technique, which can be accelerated efficiently by graphics hardware. By choosing varied basis texture sprites, waterfall scenes with different appearance and shapes can be conveniently simulated. The experimental results demonstrate that our approach achieves realistic effects and real‐time frame rates on consumer PC platforms. Copyright © 2006 John Wiley & Sons, Ltd.     

VisuaLeague: Player performance analysis using spatial-temporal data

In recent years, the phenomenon of eSports has been a growing trend and consequently, in addition to players, other groups of users, including coaches and analysts, took an interest in online video games and the data extracted from them. Among many types of video games, one of the most widely played is the MOBA (Multiplayer Online Battle Arena) League of Legend (LoL) game. Similary to traditional sports, players and coaches/analysts analyse all game events, such as, players’ movements, to understand how they play to define new strategies and improve their performance. Our main goal is to get a better understanding of which visualizations techniques are more adequate to handle this type of spatio-temporal information data, associated to player performance analysis in video games. To address this goal, we inquired players to identify the analytical questions they need to support for performance analysis and designed the VisuaLeague prototype for the visualization of in-game player trajectories, using animated maps, and events during a LoL match. This paper presents a user study to evaluate the adequacy of animated maps and the analytical strategies followed by players when using spatio-temporal data to analyse player performance. The results support the adequacy of using the animated maps technique to convey information to users in this context. Moreover, they also point out towards a high degree of importance given to the spatio-temporal components of the data for player performance analysis.

     

Transferring of speech movements from video to 3D face space

We present a novel method for transferring speech animation recorded in low quality videos to high resolution 3D face models. The basic idea is to synthesize the animated faces by an interpolation based on a small set of 3D key face shapes which span a 3D face space. The 3D key shapes are extracted by an unsupervised learning process in 2D video space to form a set of 2D visemes which are then mapped to the 3D face space. The learning process consists of two main phases: 1) isomap-based nonlinear dimensionality reduction to embed the video speech movements into a low-dimensional manifold and 2) k-means clustering in the low-dimensional space to extract 2D key viseme frames. Our main contribution is that we use the isomap-based learning method to extract intrinsic geometry of the speech video space and thus to make it possible to define the 3D key viseme shapes. To do so, we need only to capture a limited number of 3D key face models by using a general 3D scanner. Moreover, we also develop a skull movement recovery method based on simple anatomical structures to enhance 3D realism in local mouth movements. Experimental results show that our method can achieve realistic 3D animation effects with a small number of 3D key face models     

Head-Tracked Stereo Viewing with Two-Handed 3 D Interaction for Animated Character Construction

In this paper, we demonstrate how a new interactive 3 D desktop metaphor based on two-handed 3 D direct manipulation registered with head-tracked stereo viewing can be applied to the task of constructing animated characters. In our configuration, a six degree-of-freedom head-tracker and CrystalEyes shutter glasses are used to produce stereo images that dynamically follow the user head motion. 3 D virtual objects can be made to appear at a fixed location in physical space which the user may view from different angles by moving his head. To construct 3 D animated characters, the user interacts with the simulated environment using both hands simultaneously: the left hand, controlling a Spaceball, is used for 3 D navigation and object movement, while the right hand, holding a 3 D mouse, is used to manipulate through a virtual tool metaphor the objects appearing in front of the screen. In this way, both incremental and absolute interactive input techniques are provided by the system. Hand-eye coordination is made possible by registering virtual space exactly to physical space, allowing a variety of complex 3 D tasks necessary for constructing 3 D animated characters to be performed more easily and more rapidly than is possible using traditional interactive techniques. The system has been tested using both Polhemus Fastrak and Logitech ultrasonic input devices for tracking the head and 3 D mouse.     

Multi-camera video surveillance for real-time analysis and reconstruction of soccer games

Soccer analysis and reconstruction is one of the most interesting challenges for wide-area video surveillance applications. Techniques and system implementation for tracking the ball and players with multiple stationary cameras are discussed. With video data captured from a football stadium, the real-world, real-time positions of the ball and players can be generated. The whole system contains a two-stage workflow, i.e., single view and multi-view processing. The first stage includes categorizing of players and filtering of the ball after changing detection against an adaptive background and image-plane tracking. Occlusion reasoning and tracking-back is applied for robust ball filtering. In the multi-view stage, multiple observations from overlapped single views are fused to refine players’ positions and to estimate 3-D ball positions using geometric constraints. Experimental results on real data from long sequences are demonstrated.     

Multi-Object Tracking in Video

This paper reports on tracking of multiple objects using color histogram backprojection and motion cues. Four tasks which facilitate this are discussed. The first is an adaptive color histogram backprojection (which builds upon the works of Swain and Ballard) and its application to tracking of multiple objects in video sequences. The second task is designing efficient fast blob detectors for selecting regions of interest in video sequences. The third is motion detection based on color histogram backprojection. Achieving these tasks led to multi-objects tracking. Various video sequences were used to demonstrate effective tracking of multiple objects. Notably, we created an interactive multiple objects tracker (CLICK-IT) which in its present form is set at three objects but can be extended easily. CLICK-IT (CSIRO Laboratory for Imaging by Content and Knowledge—Interactive Television) is a PC-based system which provides the user with an intelligent highlighter pen for sports action replay. It is intended as a truly interactive improvement on the drawing pad technology currently used for video annotation in sports broadcasting. The system uses computer vision techniques to focus attention and track particular objects (player(s), ball, horse(s), …) and semi-automatically annotate the dynamic scene. This paper describes the system including the user interface, the tracking technology based on color and motion information, and system performance evaluation in applications to surveillance-like sequences, running, rugby league football, basketball and soccer. Finally, video scene detection based on color histogram is discussed.     

一种自动鲁棒的三维人脸重建方法

三维人脸模型已经广泛应用到视频电话、视频会议、影视制作、电脑游戏、人脸识别等多个领域。目前三维人脸建模一般使用多幅图像，且要求表情中性。本文提出了基于正、侧面任意表情三维人脸重建方法。首先对二维图像中的人脸进行特征提取，然后基于三维人脸统计模型，通过缩放、平移、旋转等方法，及全局和局部匹配，获得特定的三维人脸。基于二维图像中的人脸纹理信息，通过纹理映射，获得完整的三维人脸。通过对大量实际二维人脸图像的三维人脸重建，证实了该方法的有效性和鲁棒性。     

Photogrammetric reconstruction of free-form objects with curvilinear structures

The shapes of many natural or man-made objects have curve features. The images of such curves usually do not have sufficient distinctive features to apply conventional feature-based reconstruction algorithms. In this paper, we introduce a photogrammetric method for recovering free-form objects with curvilinear structures. Our method chooses to obtain the topology and geometry of a sparse 3D wireframe of the object first instead of directly recovering a surface or volume model. Surface patches covering the object are then constructed to interpolate the curves in this wireframe while satisfying certain heuristics such as minimal bending energy. The result is an object surface model with curvilinear structures from a sparse set of images. We can produce realistic texture-mapped renderings of the object model from arbitrary viewpoints. Reconstruction results on multiple real objects are presented to demonstrate the effectiveness of our approach.     

Interactive Construction and Animation of Layered Elastically Deformable Characters

An interactive system is described for creating and animating deformable 3D characters. By using a hybrid layered model of kinematic and physics-based components together with an immersive 3D direct manipulation interface, it is possible to quickly construct characters that deform naturally when animated and whose behavior can be controlled interactively using intuitive parameters. In this layered construction technique, called the elastic surface layer model, a simulated elastically deformable skin surface is wrapped around a kinematic articulated figure. Unlike previous layered models, the skin is free to slide along the underlying surface layers constrained by geometric constraints which push the surface out and spring forces which pull the surface in to the underlying layers. By tuning the parameters of the physics-based model, a variety of surface shapes and behaviors can be obtained such as more realistic-looking skin deformation at the joints, skin sliding over muscles, and dynamic effects such as squash-and-stretch and follow-through. Since the elastic model derives all of its input forces from the underlying articulated figure, the animator may specify all of the physical properties of the character once, during the initial character design process, after which a complete animation sequence can be created using a traditional skeleton animation technique. Character construction and animation are done using a 3D user interface based on two-handed manipulation registered with head-tracked stereo viewing. In our configuration, a six degree-of-freedom head-tracker and CrystalEyes shutter glasses are used to display stereo images on a workstation monitor that dynamically follow the user head motion. 3D virtual objects can be made to appear at a fixed location in physical space which the user may view from different angles by moving his head. To construct 3D animated characters, the user interacts with the simulated environment using both hands simultaneously: the left hand, controlling a Spaceball, is used for 3D navigation and object movement, while the right hand, holding a 3D mouse, is used to manipulate through a virtual tool metaphor the objects appearing in front of the screen. Hand-eye coordination is made possible by registering virtual space to physical space, allowing a variety of complex 3D tasks necessary for constructing 3D animated characters to be performed more easily and more rapidly than is possible using traditional interactive techniques.     

足球视频球员感知跟踪算法

目的 足球比赛视频中的球员跟踪算法为足球赛事分析提供基础的数据支持。但足球比赛中球员跟踪存在极大的挑战：球员进攻、防守和争夺球权时,目标球员可能产生快速移动、严重遮挡和周围出现若干名干扰球员的情况,目前仍没有一种能够完美解决足球比赛中球员跟踪问题的算法。因此如何解决足球场景中的困难,提升球员跟踪的准确度,成为当前研究的热点问题。方法 本文在分析足球比赛视频中球员目标特点的基础上,通过融合干扰项感知的颜色模型和目标感知的深度模型,提出并设计了一种球员感知的跟踪算法。干扰项感知的颜色模型分别提取目标、背景和干扰项的颜色直方图,利用贝叶斯公式得到搜索区域中每个像素点属于目标的似然概率。目标感知的深度模型利用孪生网络计算搜索区域与目标的相似度。针对跟踪漂移问题,使用全局跟踪器和局部跟踪器分别跟踪目标整体和目标上半身,并且在两个跟踪器的跟踪结果出现较大差异的时候分析跟踪器有效性并进行定位修正。结果 在公共的足球数据集上将本文算法与10个其他跟踪算法进行对比实验,同时对于文本算法进行了局部跟踪器的消融实验。实验结果表明,球员感知跟踪算法的平均有效重叠率达到了0.560 3,在存在同队球员和异队球员干扰的情况下,本文算法比排名第2的算法的有效重叠率分别高出3.7%和6.6%,明显优于其他算法,但是由于引入了干扰项感知的颜色模型、目标感知的深度模型以及局部跟踪器等模块增加了算法的时间复杂度,导致本文算法跟踪速度较慢。结论 本文总结了跟踪算法的整体流程并分析了实验结果,认为干扰项感知、目标感知和局部跟踪这3个策略在足球场景中的球员跟踪问题中起到了重要的作用,为未来在足球球员跟踪领域研究的继续深入提供了参考依据。     

Graph-Based Multiplayer Detection and Tracking in Broadcast Soccer Videos

In this paper, we propose a graph-based approach for detecting and tracking multiple players in broadcast soccer videos. In the first stage, the position of the players in each frame is determined by removing the non player regions. The remaining pixels are then grouped using a region growing algorithm to identify probable player candidates. A directed weighted graph is constructed, where probable player candidates correspond to the nodes of the graph while each edge links candidates in a frame with the candidates in next two consecutive frames. Finally, dynamic programming is applied to find the trajectory of each player. Experiments with several sequences from broadcasted videos of international soccer matches indicate that the proposed approach is able to track the players reasonably well even under varied illumination and ground conditions.      

多摄像机定标恢复足球比赛场景

在足球运动中,分析比赛需要了解足球场与运动员以及足球之间的位置关系.DLT定标是一种常用的算法,通过改进这种摄像机定标算法,从有限的比赛图像中恢复部分摄像机参数,计算出足球运动员相对于足球场的坐标.实验由两幅静态的图像完成了摄像机定标,通过定标参数和图像对运动员的三维坐标进行了恢复,得到了运动员的三维棍图.     

误差云映射的深度融合3维重建

描述了一种应用于视频序列的3维稠密重建方法,主要针对基于多深度图恢复3维物体时存在的多对一映射重复投影问题,提出基于误差云概念的3维定点优化方法.该方法首先通过深度矫正,增加视频序列深度图的一致性;其次利用投影过滤分类算法,把所有投影点按照多对一映射的关系,以3维空间中所有不同点为类型,进行投影点依次映射,将每一点划分在各自的误差云类中以求得投影点与3维点的对应关系;随后采用空间高斯分布求出每个误差云所恢复出来的点坐标.最后通过多边形技术对恢复的点云数据进行网格化,使其重建出精确的目标物体或场景的3维轮廓.从实验结果可以观察到,本文3维重建方法可以有效减少深度图融合多对一映射重复投影问题所带来的负面影响,使重建结果更加接近于真实物体,达到较好的效果.     

3D Z-string: A new knowledge structure to represent spatio-temporal relations between objects in a video

In this paper, we propose a new knowledge structure called 3D Z-string, extended from the 2D Z-string, to represent the spatial and temporal relations between objects in a video and to keep track of the motions and size changes of the objects. Since there are no cuttings between objects in the 3D Z-string, the integrity of objects is preserved. The string generation and video reconstruction algorithms for the 3D Z-string representation of video objects are also developed. The string generated by the string generation algorithm is unique for a given video and the video reconstructed from a given 3D Z-string is unique too. The experimental results show that the 3D Z-string is more compact and efficient than the 3D C-string in terms of storage space and execution time.     

Ball tracking and 3D trajectory approximation with applications to tactics analysis from single-camera volleyball sequences

Providing computer-assisted tactics analysis in sports is a growing trend. This paper presents an automatic system for ball tracking and 3D trajectory approximation from single-camera volleyball sequences as well as demonstrates several applications to tactics analysis. Ball tracking in volleyball video has great complexity due to the high density of players on the court and the complicated overlapping of ball-player. The 2D-to-3D inference is intrinsically challenging due to the loss of 3D information in projection to 2D frames. To overcome these challenges, we propose a two-phase ball tracking algorithm in which we first detect ball candidates for each frame, and then use them to compute the ball trajectories. With the aid of camera calibration, we involve physical characteristics of ball motion to approximate the 3D ball trajectory from the 2D trajectory. The visualization of 3D trajectory and the applications to trajectory-based tactics analysis not only assist the coaches and players in game study but also make game watching a whole new experience. The experiments on international volleyball games show encouraging results. We believe that the proposed framework can be extended and applied to various kinds of sports games.     

Bayesian belief network based broadcast sports video indexing

This paper presents a probabilistic Bayesian belief network (BBN) method for automatic indexing of excitement clips of sports video sequences. The excitement clips from sports video sequences are extracted using audio features. The excitement clips are comprised of multiple subclips corresponding to the events such as replay, field-view, close-ups of players, close-ups of referees/umpires, spectators, players’ gathering. The events are detected and classified using a hierarchical classification scheme. The BBN based on observed events is used to assign semantic concept-labels to the excitement clips, such as goals, saves, and card in soccer video, wicket and hit in cricket video sequences. The BBN based indexing results are compared with our previously proposed event-association based approach and found BBN is better than the event-association based approach. The proposed scheme provides a generalizable method for linking low-level video features with high-level semantic concepts. The generic nature of the proposed approach in the sports domain is validated by demonstrating successful indexing of soccer and cricket video excitement clips. The proposed scheme offers a general approach to the automatic tagging of large scale multimedia content with rich semantics. The collection of labeled excitement clips provide a video summary for highlight browsing, video skimming, indexing and retrieval.