People counting via multiple views using a fast information fusion approach

Real-time estimates of a crowd size is a central task in civilian surveillance. In this paper we present a novel system counting people in a crowd scene with overlapping cameras. This system fuses all single view foreground information to localize each person present on the scene. The purpose of our fusion strategy is to use the foreground pixels of each single views to improve real-time objects association between each camera of the network. The foreground pixels are obtained by using an algorithm based on codebook. In this work, we aggregate the resulting silhouettes over cameras network, and compute a planar homography projection of each camera’s visual hull into ground plane. The visual hull is obtained by finding the convex hull of the foreground pixels. After the projection into the ground plane, we fuse the obtained polygons by using the geometric properties of the scene and on the quality of each camera detection. We also suggest a region-based approach tracking strategy which keeps track of people movements and of their identities along time, also enabling tolerance to occasional misdetections. This tracking strategy is implemented on the result of the views fusion and allows to estimate the crowd size dependently on each frame. Assessment of experiments using public datasets proposed for the evaluation of counting people system demonstrates the performance of our fusion approach. These results prove that the fusion strategy can run in real-time and is efficient for making data association. We also prove that the combination of our fusion approach and the proposed tracking improve the people counting.

     

Robust camera pose tracking for augmented reality using particle filtering framework

In this paper, we present new solutions for the problem of estimating the camera pose using particle filtering framework. The proposed approach is suitable for real-time augmented reality (AR) applications in which the camera is held by the user. This work demonstrates that particle filtering improve the robustness of the tracking comparing to existing approaches, such as those based on the Kalman filter. We propose a tracking framework for both points and lines features, the particle filter is used to compute the posterior density for the camera 3D motion parameters. We also analyze the sensitivity of our technique when outliers are present in the match data. Outliers arise frequently due to incorrect correspondences which occur because of either image noise or occlusion. Results from real data in an augmented reality setup are then presented, demonstrating the efficiency and robustness of the proposed method.     

基于云台摄像的实时车速检测算法

针对采用固定摄像的路况监视系统无法观看自如的缺点,提出了基于云台摄像的实时车速检测算法．建立了简化的摄像机参数模型,提取了线性拟合后的车道图像特征参数,并利用Kluge曲线模型和随机霍夫变换实现了像平面车道分割线的二维重建和云台摄像机的标定;应用自适应背景减除、扩展Kalman滤波器等方法,提取了帧运动域及域中目标轮廓,从而实现了车辆的精确定位、跟踪,以至实时速度检测．该算法已试用于工程实践,具有较好的鲁棒性．     

Kinect based real-time synthetic aperture imaging through occlusion

Real-time and high performance occluded object imaging is a big challenge to many computer vision applications. In recent years, camera array synthetic aperture theory proves to be a potential powerful way to solve this problem. However, due to the high cost of complex system hardware, the severe blur of occluded object imaging, and the slow speed of image processing, the exiting camera array synthetic aperture imaging algorithms and systems are difficult to apply in practice. In this paper, we present a novel handheld system to handle those challenges. The objective of this work is to design a convenient system for real-time high quality object imaging even under severe occlusion. The main characteristics of our work include: (1) To the best of our knowledge, this is the first real-time handheld system for seeing occluded object in synthetic imaging domain using color and depth images. (2) A novel sequential synthetic aperture imaging framework is designed to achieve seamless interaction among multiple novel modules, and this framework includes object probability generation, virtual camera array generation, and sequential synthetic aperture imaging. (3) In the virtual camera array generation module, based on the integration of color and depth information, a novel feature set iterative optimization algorithm is presented, which can improve the robustness and accuracy of camera pose estimation even in dynamic occlusion scene. Experimental results in challenging scenarios demonstrate the superiority of our system both in robustness and efficiency compared against the state-of-the-art algorithms.     

Person re-identification with block sparse recovery

We consider the problem of automatically re-identifying a person of interest seen in a “probe” camera view among several candidate people in a “gallery” camera view. This problem, called person re-identification, is of fundamental importance in several video analytics applications. While extracting knowledge from high-dimensional visual representations based on the notions of sparsity and regularization has been successful for several computer vision problems, such techniques have not been fully exploited in the context of the re-identification problem. Here, we develop a principled algorithm for the re-identification problem in the general framework of learning sparse visual representations. Given a set of feature vectors for a person in one camera view (corresponding to multiple images as they are tracked), we show that a feature vector representing the same person in another view approximately lies in the linear span of this feature set. Furthermore, under certain conditions, the associated coefficient vector can be characterized as being block sparse. This key insight allows us to design an algorithm based on block sparse recovery that achieves state-of-the-art results in multi-shot person re-identification. We also revisit an older feature transformation technique, Fisher discriminant analysis, and show that, when combined with our proposed formulation, it outperforms many sophisticated methods. Additionally, we show that the proposed algorithm is flexible and can be used in conjunction with existing metric learning algorithms, resulting in improved ranking performance. We perform extensive experiments on several publicly available datasets to evaluate the proposed algorithm.     

Algorithms for coplanar camera calibration

Abstract. Coplanar camera calibration is the process of determining the extrinsic and intrinsic camera parameters from a given set of image and world points, when the world points lie on a two-dimensional plane. Noncoplanar calibration, on the other hand, involves world points that do not lie on a plane. While optimal solutions for both the camera-calibration procedures can be obtained by solving a set of constrained nonlinear optimization problems, there are significant structural differences between the two formulations. We investigate the computational and algorithmic implications of such underlying differences, and provide a set of efficient algorithms that are specifically tailored for the coplanar case. More specifically, we offer the following: (1) four algorithms for coplanar calibration that use linear or iterative linear methods to solve the underlying nonlinear optimization problem, and produce sub-optimal solutions. These algorithms are motivated by their computational efficiency and are useful for real-time low-cost systems. (2) Two optimal solutions for coplanar calibration, including one novel nonlinear algorithm. A constraint for the optimal estimation of extrinsic parameters is also given. (3) A Lyapunov type convergence analysis for the new nonlinear algorithm. We test the validity and performance of the calibration procedures with both synthetic and real images. The results consistently show significant improvements over less complete camera models. Received: 30 September 1998 / Accepted: 12 January 2000     

Automatic camera calibration of broadcast tennis video with applications to 3D virtual content insertion and ball detection and tracking

This paper presents an original algorithm to automatically acquire accurate camera calibration from broadcast tennis video (BTV) as well as demonstrates two of its many applications. Accurate camera calibration from BTV is challenging because the frame-data of BTV is often heavily distorted and full of errors, resulting in wildly fluctuating camera parameters. To meet this challenge, we propose a frame grouping technique, which is based on the observation that many frames in BTV possess the same camera viewpoint. Leveraging on this fact, our algorithm groups frames according to the camera viewpoints. We then perform a group-wise data analysis to obtain a more stable estimate of the camera parameters. Recognizing the fact that some of these parameters do vary somewhat even if they have similar camera viewpoint, we further employ a Hough-like search to tune such parameters, minimizing the reprojection disparity. This two-tiered process gains stability in the estimates of the camera parameters, and yet ensures good match between the model and the reprojected camera view via the tuning step. To demonstrate the utility of such stable calibration, we apply the camera matrix acquired to two applications: (a) 3D virtual content insertion; and (b) tennis-ball detection and tracking. The experimental results show that our algorithm is able to acquire accurate camera matrix and the two applications have very good performances.     

Head-Size Equalization for Improved Visual Perception in Video Conferencing

In a video conferencing setting, people often use an elongated meeting table with the major axis along the camera direction. A standard wide-angle perspective image of this setting creates significant foreshortening, thus the people sitting at the far end of the table appear very small relative to those nearer the camera. This has two consequences. First, it is difficult for the remote participants to see the faces of those at the far end, thus affecting the experience of the video conferencing. Second, it is a waste of the screen space and network bandwidth because most of the pixels are used on the background instead of on the faces of the meeting participants. In this paper, we present a novel technique, called Spatially-Varying-Uniform scaling functions, to warp the images to equalize the head sizes of the meeting participants without causing undue distortion. This technique works for both the 180-degree views where the camera is placed at one end of the table and the 360-degree views where the camera is placed at the center of the table. We have implemented this algorithm on two types of camera arrays: one with 180-degree view, and the other with 360-degree view. On both hardware devices, image capturing, stitching, and head-size equalization are run in real time. In addition, we have conducted user study showing that people clearly prefer head-size equalized images.     

ClariSense+: An enhanced traffic anomaly explanation service using social network feeds

The explosive growth in social networks that publish real-time content begs the question of whether their feeds can complement traditional sensors to achieve augmented sensing capabilities. One such capability is to explain anomalous sensor readings. In our previous conference paper, we built an automated anomaly clarification service, called ClariSense, with the ability to explain sensor anomalies using social network feeds (from Twitter). In this extended work, we present an enhanced anomaly explanation system that augments our base algorithm by considering both (i) the credibility of social feeds and (ii) the spatial locality of detected anomalies. The work is geared specifically for describing small-footprint anomalies, such as vehicular traffic accidents. The original system used information gain to select more informative microblog items to explain physical sensor anomalies. In this paper, we show that significant improvements are achieved in our ability to explain small-footprint anomalies by accounting for information credibility and further discriminating among high-information-gain items according to the size of their spatial footprint. Hence, items that lack sufficient corroboration and items whose spatial footprint in the blogosphere is not specific to the approximate location of the physical anomaly receive less consideration. We briefly demonstrate the workings of such a system by considering a variety of real-world anomalous events, and comparing their causes, as identified by ClariSense+, to ground truth for validation. A more systematic evaluation of this work is done using vehicular traffic anomalies. Specifically, we consider real-time traffic flow feeds shared by the California traffic system. When flow anomalies are detected, our system automatically diagnoses their root cause by correlating the anomaly with feeds on Twitter. For evaluation purposes, the identified cause is then retroactively compared to official traffic and incident reports that we take as ground truth. Results show a great correspondence between our automatically selected explanations and ground-truth data.     

Beyond contracts for concurrency

SCOOP is a concurrent programming language with a new semantics for contracts that applies equally well in concurrent and sequential contexts. SCOOP eliminates race conditions and atomicity violations by construction. However, it is still vulnerable to deadlocks. In this paper we describe how far contracts can take us in verifying interesting properties of concurrent systems using modular Hoare rules and show how theorem proving methods developed for sequential Eiffel can be extended to the concurrent case. However, some safety and liveness properties depend upon the environment and cannot be proved using the Hoare rules. To deal with such system properties, we outline a SCOOP Virtual Machine (SVM) as a fair transition system. The SVM makes it feasible to use model-checking and theorem proving methods for checking global temporal logic properties of SCOOP programs. The SVM uses the Hoare rules where applicable to reduce the number of steps in a computation. P. J. Brooke, R. F. Paige and Dong Jin Song This work was conducted under an NSERC Discovery grant.     

ClariSense+: An enhanced traffic anomaly explanation service using social network feeds

The explosive growth in social networks that publish real-time content begs the question of whether their feeds can complement traditional sensors to achieve augmented sensing capabilities. One such capability is to explain anomalous sensor readings. In our previous conference paper, we built an automated anomaly clarification service, called ClariSense, with the ability to explain sensor anomalies using social network feeds (from Twitter). In this extended work, we present an enhanced anomaly explanation system that augments our base algorithm by considering both (i) the credibility of social feeds and (ii) the spatial locality of detected anomalies. The work is geared specifically for describing small-footprint anomalies, such as vehicular traffic accidents. The original system used information gain to select more informative microblog items to explain physical sensor anomalies. In this paper, we show that significant improvements are achieved in our ability to explain small-footprint anomalies by accounting for information credibility and further discriminating among high-information-gain items according to the size of their spatial footprint. Hence, items that lack sufficient corroboration and items whose spatial footprint in the blogosphere is not specific to the approximate location of the physical anomaly receive less consideration. We briefly demonstrate the workings of such a system by considering a variety of real-world anomalous events, and comparing their causes, as identified by ClariSense+, to ground truth for validation. A more systematic evaluation of this work is done using vehicular traffic anomalies. Specifically, we consider real-time traffic flow feeds shared by the California traffic system. When flow anomalies are detected, our system automatically diagnoses their root cause by correlating the anomaly with feeds on Twitter. For evaluation purposes, the identified cause is then retroactively compared to official traffic and incident reports that we take as ground truth. Results show a great correspondence between our automatically selected explanations and ground-truth data.     

Real-Time Camera Calibration for Virtual Studio

In this paper, we present an overall algorithm for real-time camera parameter extraction, which is one of the key elements in implementing virtual studio, and we also present a new method for calculating the lens distortion parameter in real time. In a virtual studio, the motion of a virtual camera generating a graphic studio must follow the motion of the real camera in order to generate a realistic video product. This requires the calculation of camera parameters in real-time by analyzing the positions of feature points in the input video. Towards this goal, we first design a special calibration pattern utilizing the concept of cross-ratio, which makes it easy to extract and identify feature points, so that we can calculate the camera parameters from the visible portion of the pattern in real-time. It is important to consider the lens distortion when zoom lenses are used because it causes nonnegligible errors in the computation of the camera parameters. However, the Tsai algorithm, adopted for camera calibration, calculates the lens distortion through nonlinear optimization in triple parameter space, which is inappropriate for our real-time system. Thus, we propose a new linear method by calculating the lens distortion parameter independently, which can be computed fast enough for our real-time application. We implement the whole algorithm using a Pentium PC and Matrox Genesis boards with five processing nodes in order to obtain the processing rate of 30 frames per second, which is the minimum requirement for TV broadcasting. Experimental results show this system can be used practically for realizing a virtual studio.     

Floating Textures

We present a novel multi‐view, projective texture mapping technique. While previous multi‐view texturing approaches lead to blurring and ghosting artefacts if 3D geometry and/or camera calibration are imprecise, we propose a texturing algorithm that warps (“floats”) projected textures during run‐time to preserve crisp, detailed texture appearance. Our GPU implementation achieves interactive to real‐time frame rates. The method is very generally applicable and can be used in combination with many image‐based rendering methods or projective texturing applications. By using Floating Textures in conjunction with, e.g., visual hull rendering, light field rendering, or free‐viewpoint video, improved rendering results are obtained from fewer input images, less accurately calibrated cameras, and coarser 3D geometry proxies.     

Contour tracking with automatic motion model switching

In this paper we present an efficient contour-tracking algorithm which can track 2D silhouette of objects in extended image sequences. We demonstrate the ability of the tracker by tracking highly deformable contours (such as walking people) captured by a static camera. We represent contours (silhouette) of moving objects by using a cubic B-spline. The tracking algorithm is based on tracking a lower dimensional shape space (as opposed to tracking in spline space). Tracking the lower dimensional space has proved to be fast and efficient. The tracker is also coupled with an automatic motion-model switching algorithm, which makes the tracker robust and reliable when the object of interest is moving with multiple motion. The model-based tracking technique provided is capable of tracking rigid and non-rigid object contours with good tracking accuracy.     

Backpack: Detection of People Carrying Objects Using Silhouettes

We describe a video-rate surveillance algorithm for determining whether people are carrying objects or moving unencumbered from a stationary camera. The contribution of the paper is the shape analysis algorithm that both determines whether a person is carrying an object and segments the object from the person so that it can be tracked, e.g., during an exchange of objects between two people. As the object is segmented, an appearance model of the object is constructed. The method combines periodic motion estimation with static symmetry analysis of the silhouettes of a person in each frame of the sequence. Experimental results demonstrate robustness and real-time performance of the proposed algorithm.     

Hybrid evolutionary algorithm for microscrew thread parameter estimation

In this paper we propose a novel approach to the problem of microscrew thread parameter estimation based on a hybrid evolutionary algorithm that combines a stochastic evolutionary algorithm with the deterministic inverse parabolic interpolation. The proposed method uses a machine vision system utilizing a single web camera. The hybrid evolutionary algorithm was tested on a specially created image database of microscrews. Experimental results prove speed and efficiency of the proposed method and its robustness to noise in the images. This method may be used in automated systems of real-time non-destructive quality control of microscrews and has potential for parameter estimation of different types of microparts.     

Stable real-time 3D tracking using online and offline information

We propose an efficient real-time solution for tracking rigid objects in 3D using a single camera that can handle large camera displacements, drastic aspect changes, and partial occlusions. While commercial products are already available for offline camera registration, robust online tracking remains an open issue because many real-time algorithms described in the literature still lack robustness and are prone to drift and jitter. To address these problems, we have formulated the tracking problem in terms of local bundle adjustment and have developed a method for establishing image correspondences that can equally well handle short and wide-baseline matching. We then can merge the information from preceding frames with that provided by a very limited number of keyframes created during a training stage, which results in a real-time tracker that does not jitter or drift and can deal with significant aspect changes.     

A new approach to real-time mosaicing of aerial images

We present a new image mosaicing technique that uses sequential aerial images captured from a camera and is capable of creating consistent large scale mosaics in real-time. To find the alignment of every new image, we use all the available images in the mosaic that have intersection with the new image instead of using only the previous one. To detect image intersections in an efficient manner, we utilize ‘Separating Axis Theorem’, a geometric tool from computer graphics which is used for collision detection. Moreover, after a certain number of images are added to the mosaic, a novel affine refinement procedure is carried out to increase global consistency. Finally, gain compensation and multi-band blending are optionally used as offline steps to compensate for photometric defects and seams caused by misregistrations. Proposed approach is tested on some public datasets and it is compared with two state-of-the-art algorithms. Results are promising and show the potential of our algorithm in various practical scenarios.     

MonoSLAM: real-time single camera SLAM

We present a real-time algorithm which can recover the 3D trajectory of a monocular camera, moving rapidly through a previously unknown scene. Our system, which we dub MonoSLAM, is the first successful application of the SLAM methodology from mobile robotics to the "pure vision" domain of a single uncontrolled camera, achieving real time but drift-free performance inaccessible to structure from motion approaches. The core of the approach is the online creation of a sparse but persistent map of natural landmarks within a probabilistic framework. Our key novel contributions include an active approach to mapping and measurement, the use of a general motion model for smooth camera movement, and solutions for monocular feature initialization and feature orientation estimation. Together, these add up to an extremely efficient and robust algorithm which runs at 30 Hz with standard PC and camera hardware. This work extends the range of robotic systems in which SLAM can be usefully applied, but also opens up new areas. We present applications of MonoSLAM to real-time 3D localization and mapping for a high-performance full-size humanoid robot and live augmented reality with a hand-held camera