Camera handoff and placement for automated tracking systems with multiple omnidirectional cameras

In a multi-camera surveillance system, both camera handoff and placement play an important role in generating an automated and persistent object tracking, typical of most surveillance requirements. Camera handoff should comprise three fundamental components, time to trigger handoff process, the execution of consistent labeling, and the selection of the next optimal camera. In this paper, we design an observation measure to quantitatively formulate the effectiveness of object tracking so that we can trigger camera handoff timely and select the next camera appropriately before the tracked object falls out of the field of view (FOV) of the currently observing camera. In the meantime, we present a novel solution to the consistent labeling problem in omnidirectional cameras. A spatial mapping procedure is proposed to consider both the noise inherent to the tracking algorithms used by the system and the lens distortion introduced by omnidirectional cameras. This does not only avoid the tedious process, but also increases the accuracy, to obtain the correspondence between omnidirectional cameras without human interventions. We also propose to use the Wilcoxon Signed-Rank Test to improve the accuracy of trajectory association between pairs of objects. In addition, since we need a certain amount of time to successfully carry out the camera handoff procedure, we introduce an additional constraint to optimally reserve sufficient cameras’ overlapped FOVs for the camera placement. Experiments show that our proposed observation measure can quantitatively formulate the effectiveness of tracking, so that camera handoff can smoothly transfer objects of interest. Meanwhile, our proposed consistent labeling approach can perform as accurately as the geometry-based approach without tedious calibration processes and outperform Calderara’s homography-based approach. Our proposed camera placement method exhibits a significant increase in the camera handoff success rate at the cost of slightly decreased coverage, as compared to Erdem and Sclaroff’s method without considering the requirement on overlapped FOVs.     

Camera array calibration for light field acquisition

Light field cameras are becoming popular in computer vision and graphics, with many research and commercial applications already having been proposed.Various types of cameras have been developed with the camera array being one of the ways of acquiring a 4D light field image usingmultiple cameras. Camera calibration is essential, since each application requires the correct projection and ray geometry of the light field. The calibrated parameters are used in the light field image rectified from the images captured by multiple cameras. Various camera calibration approaches have been proposed for a single camera, multiple cameras, and amoving camera. However, although these approaches can be applied to calibrating camera arrays, they are not effective in terms of accuracy and computational cost. Moreover, less attention has been paid to camera calibration of a light field camera. In this paper, we propose a calibration method for a camera array and a rectification method for generating a light field image from the captured images. We propose a two-step algorithm consisting of closed form initialization and nonlinear refinement, which extends Zhang’swell-known method to the camera array. More importantly, we introduce a rigid camera constraint whereby the array of cameras is rigidly aligned in the camera array and utilize this constraint in our calibration. Using this constraint, we obtained much faster and more accurate calibration results in the experiments.     

Bus surveillance: how many and where cameras should be placed

Public transport safety is an important issue that has recently gained substantial attention, especially with the increasing number of violent incidents occurring abroad. To avoid such incidents and to perform post-incident investigations, many buses today are equipped with surveillance cameras. These cameras are usually installed in key places such as doors, the front and the middle of the bus. This camera placement is often performed manually based on human intuition and knowledge; however, there is no scientific basis to justify: (1) how many cameras would be sufficient, and (2) where (location) and how (with what orientation) they should be placed, to increase the area of coverage at the minimum cost. This paper addresses this issue by breaking it down into two separate problems: MaxGain and MinCost. The MaxGain problem is aimed to maximize the overall coverage with a specific number of cameras; while the MinCost problem attempts to minimize the number of cameras to cover a specified area in the bus. The solutions to these two problems are presented. The proposed method computes the approximate coverage of a camera inside the 3D bus model. Furthermore, in order to improve the efficiency of the solution, an algorithm called “SmartMax” is proposed. The proposed solution advises precise locations and orientations (pan and tilt angles) of required cameras and can be used to validate the current camera installations in various types of public transit buses.     

Target coverage in camera networks for manufacturing workplaces

In this paper, we investigate the camera network placement problem for target coverage in manufacturing workplaces. The problem is formulated to find the minimum number of cameras of different types and their best configurations to maximise the coverage of the monitored workplace such that the given set of target points of interest are each k-covered with a predefined minimum spatial resolution. Since the problem is NP-complete, and even NP-hard to approximate, a novel method based on Simulated Annealing is presented to solve the optimisation problem. A new neighbourhood generation function is proposed to handle the discrete nature of the problem. The visual coverage is modelled using realistic and coherent assumptions of camera intrinsic and extrinsic parameters making it suitable for many real world camera based applications. Task-specific quality of coverage measure is proposed to assist selecting the best among the set of camera network placements with equal coverage. A 3D CAD of the monitored space is used to examine physical occlusions of target points. The results show the accuracy, efficiency and scalability of the presented solution method; which can be applied effectively in the design of practical camera networks.     

Optimal placement of multiple visual sensors considering space coverage and cost constraints

This paper proposes an optimal camera placement method that analyzes static spatial information in various aspects and calculates priorities of spaces using modeling the moving people pattern and simulation of pedestrian movement. To derive characteristics of space and to cover the space efficiently, an agent-based camera placement method has been developed considering the camera performance as well as the space utility extracted from a path finding algorithm. The simulation shows that the method not only determines the optimal number of cameras, but also coordinates the position and orientation of a camera efficiently considering the installation costs. Experimental results show that our approach achieves a great performance enhancement compared to other existing methods.     

Can You See Me Now? Sensor Positioning for Automated and Persistent Surveillance

Most existing camera placement algorithms focus on coverage and/or visibility analysis, which ensures that the object of interest is visible in the camera's field of view (FOV). However, visibility, which is a fundamental requirement of object tracking, is insufficient for automated persistent surveillance. In such applications, a continuous consistently labeled trajectory of the same object should be maintained across different camera views. Therefore, a sufficient uniform overlap between the cameras' FOVs should be secured so that camera handoff can successfully and automatically be executed before the object of interest becomes untraceable or unidentifiable. In this paper, we propose sensor-planning methods that improve existing algorithms by adding handoff rate analysis. Observation measures are designed for various types of cameras so that the proposed sensor-planning algorithm is general and applicable to scenarios with different types of cameras. The proposed sensor-planning algorithm preserves necessary uniform overlapped FOVs between adjacent cameras for an optimal balance between coverage and handoff success rate. In addition, special considerations such as resolution and frontal-view requirements are addressed using two approaches: 1) direct constraint and 2) adaptive weights. The resulting camera placement is compared with a reference algorithm published by Erdem and Sclaroff. Significantly improved handoff success rates and frontal-view percentages are illustrated via experiments using indoor and outdoor floor plans of various scales.      

Pose estimation from multiple cameras based on Sylvester’s equation

In this paper, we introduce a method to estimate the object’s pose from multiple cameras. We focus on direct estimation of the 3D object pose from 2D image sequences. Scale-Invariant Feature Transform (SIFT) is used to extract corresponding feature points from adjacent images in the video sequence. We first demonstrate that centralized pose estimation from the collection of corresponding feature points in the 2D images from all cameras can be obtained as a solution to a generalized Sylvester’s equation. We subsequently derive a distributed solution to pose estimation from multiple cameras and show that it is equivalent to the solution of the centralized pose estimation based on Sylvester’s equation. Specifically, we rely on collaboration among the multiple cameras to provide an iterative refinement of the independent solution to pose estimation obtained for each camera based on Sylvester’s equation. The proposed approach to pose estimation from multiple cameras relies on all of the information available from all cameras to obtain an estimate at each camera even when the image features are not visible to some of the cameras. The resulting pose estimation technique is therefore robust to occlusion and sensor errors from specific camera views. Moreover, the proposed approach does not require matching feature points among images from different camera views nor does it demand reconstruction of 3D points. Furthermore, the computational complexity of the proposed solution grows linearly with the number of cameras. Finally, computer simulation experiments demonstrate the accuracy and speed of our approach to pose estimation from multiple cameras.     

Fast Relative Depth Computation for an Active Stereo Vision System

Real-time computation of exact depth is not feasible in an active vision setup. Instead, reliable relative depth information which can be rapidly computed is preferred. In this paper, a stereo cue for computing relative depth obtained from an active stereo vision system is proposed. The proposed stereo cue can be computed purely from the coordinates of points in the stereo pair. The computational cost required is very low. No camera calibration or prior knowledge of the parameters of the stereo vision system is required. We show that the relationship between the relative depth cue and the actual depth in the three-dimensional (3D) space is monotonic. Such a relation is maintained even when the focal length and the vergence angle are changed, so long as the focal lengths of the two cameras are similar. Therefore, real-time implementation in an active vision setup can be realized. Stability analysis shows that the proposed method will be stable in practical situations, unless the stereo camera diverges. Experimental results are presented to highlight the properties and advantages of the proposed method.     

Method for HDR-like imaging using industrial digital cameras

Dynamic range of the scene can be significantly wider than the dynamic range of an image because of limitations of A/D conversion. In such a situation, numerous details of the scene cannot be adequately shown on the image. Standard industrial digital cameras are equipped with an auto-exposure function that automatically sets both the aperture value and cameras exposure time. When measuring a scene with atypical distribution of light and dark elements, the indicated auto-exposure time may not be optimal. The aim of work was to improve, with minimal cost, the performance of standard industrial digital cameras. We propose a low complexity method for creating HDR-like image using three images captured with different exposure times. The proposed method consists of three algorithms: (1) algorithm for estimating whether the auto-exposure time is optimal, (2) algorithm which determines exposure times for two additional images (one with shorter and another with longer than auto-exposure time), and (3) algorithm for HDR-like imaging based on fusion of three previously obtained images. Method is implemented on FPGA inserted into standard industrial digital camera. Results show that the proposed approach produces high quality HDR-like scene-mapped 8-bit images with minimal computational cost. All improvements may be noticed through the performance evaluation.

     

Effective demosaicking algorithm based on edge property for color filter arrays

Under the influence of digitization, digital still cameras (DSC) are becoming prevalent in recent years. The image capturing and file storing process of a digital still camera involves multiple image processing and precise corrective calculations. In order to reduce cost and volume, a digital still camera usually utilizes only one sensor and color filter array (CFA) for capturing images, and reconstructs a corresponding full-color image using a color interpolation method. Demosaicking is the first step of image processing of digital still cameras and has been integrated into the design of a variety of digital still cameras. If noise and blurred edges exist from the onset of image reconstruction, a post-processing can do little to improve the quality of the reconstructed image. A demosaicking method is proposed in this paper to prevent the occurrence of color artifacts. By detecting the edge characteristics of a digital image, accurate weights can be obtained for image interpolation, before refinement is made in post-processing. After comparing the experiment results of this paper with those of previously proposed methods, it is found that the proposed method can effectively reduce color artifacts and enhance image quality.     

Optimal Camera Placement for Automated Surveillance Tasks

Camera placement has an enormous impact on the performance of vision systems, but the best placement to maximize performance depends on the purpose of the system. As a result, this paper focuses largely on the problem of task-specific camera placement. We propose a new camera placement method that optimizes views to provide the highest resolution images of objects and motions in the scene that are critical for the performance of some specified task (e.g. motion recognition, visual metrology, part identification, etc.). A general analytical formulation of the observation problem is developed in terms of motion statistics of a scene and resolution of observed actions resulting in an aggregate observability measure. The goal of this system is to optimize across multiple cameras the aggregate observability of the set of actions performed in a defined area. The method considers dynamic and unpredictable environments, where the subject of interest changes in time. It does not attempt to measure or reconstruct surfaces or objects, and does not use an internal model of the subjects for reference. As a result, this method differs significantly in its core formulation from camera placement solutions applied to problems such as inspection, reconstruction or the Art Gallery class of problems. We present tests of the system’s optimized camera placement solutions using real-world data in both indoor and outdoor situations and robot-based experimentation using an all terrain robot vehicle-Jr robot in an indoor setting.     

Camera cooperation for achieving visual attention

In this paper we address the problem of establishing a computational model for visual attention using cooperation between two cameras. More specifically we wish to maintain a visual event within the field of view of a rotating and zooming camera through the understanding and modeling of the geometric and kinematic coupling between a static camera and an active camera. The static camera has a wide field of view thus allowing panoramic surveillance at low resolution. High-resolution details may be captured by a second camera, provided that it looks in the right direction. We derive an algebraic formulation for the coupling between the two cameras and we specify the practical conditions yielding a unique solution. We describe a method for separating a foreground event (such as a moving object) from its background while the camera rotates. A set of outdoor experiments shows the two-camera system in operation.     

一种无标记的身体与面部运动同步捕获方法

提供了一个无标记点的身体与面部运动同步捕获的方法.利用经过时间同步和空间标定的长焦彩色相机和Kinect相机来进行同步捕获.利用在环境中加入闪光来进行时间同步,使用张氏标定法进行空间标定,从而组成一组时间同步且空间对齐的混合相机（hybrid camera）.然后利用Kinect fusion扫描用户的人体模型并嵌入骨骼.最后利用时间和空间都对齐好的两个相机来进行同步采集.首先从深度图像中得到人脸的平移参考值,然后在平移参考值的帮助下根据彩色图像的2D特征点重建人脸.随后,把彩色图像中得到的头部姿态传递给身体捕获结果.结果对比实验和用户调研实验均表明所提出的运动捕获的结果要好于单个的运动捕获结果.     

Position-Based Visual Servoing in Industrial Multirobot Cells Using a Hybrid Camera Configuration

This paper deals with the problem of position-based visual servoing in a multiarm robotic cell equipped with a hybrid eye-in-hand/eye-to-hand multicamera system. The proposed approach is based on the real-time estimation of the pose of a target object by using the extended Kalman filter. The data provided by all the cameras are selected by a suitable algorithm on the basis of the prediction of the object self-occlusions, as well as of the mutual occlusions caused by the robot links and tools. Only an optimal subset of image features is considered for feature extraction, thus ensuring high estimation accuracy with a computational cost independent of the number of cameras. A salient feature of the paper is the implementation of the proposed approach to the case of a robotic cell composed of two industrial robot manipulators. Two different case studies are presented to test the effectiveness of the hybrid camera configuration and the robustness of the visual servoing algorithm with respect to the occurrence of occlusions     

Ultra-fast Lensless Computational Imaging through 5D Frequency Analysis of Time-resolved Light Transport

Light transport has been analyzed extensively, in both the primal domain and the frequency domain. Frequency analyses often provide intuition regarding effects introduced by light propagation and interaction with optical elements; such analyses encourage optimal designs of computational cameras that efficiently capture tailored visual information. However, previous analyses have relied on instantaneous propagation of light, so that the measurement of the time dynamics of light–scene interaction, and any resulting information transfer, is precluded. In this paper, we relax the common assumption that the speed of light is infinite. We analyze free space light propagation in the frequency domain considering spatial, temporal, and angular light variation. Using this analysis, we derive analytic expressions for information transfer between these dimensions and show how this transfer can be exploited for designing a new lensless imaging system. With our frequency analysis, we also derive performance bounds for the proposed computational camera architecture and provide a mathematical framework that will also be useful for future ultra-fast computational imaging systems.     

基于模式噪声大分量的手机相机来源检测

将模式噪声作为固定指纹特征应用于手机相机来源检测时,存在计算复杂、效率不高等问题。为此,提出一种基于模式噪声大分量信息的手机相机来源检测方法。利用光响应敏感点的成像特性及对模式噪声的影响,将其从模式噪声中分离出来,构成模式噪声的大分量信息,并作为新的模式噪声进行手机相机来源检测。实验结果表明,与传统基于模式噪声的图像来源检测方法相比,该方法不仅能有效辨识手机相机图像的来源,而且能减少检测计算量。     

Baseline Detection and Localization for Invisible Omnidirectional Cameras

Two key problems for camera networks that observe wide areas with many distributed cameras are self-localization and camera identification. Although there are many methods for localizing the cameras, one of the easiest and most desired methods is to estimate camera positions by having the cameras observe each other; hence the term self-localization. If the cameras have a wide viewing field, e.g. an omnidirectional camera, and can observe each other, baseline distances between pairs of cameras and relative locations can be determined. However, if the projection of a camera is relatively small on the image of other cameras and is not readily visible, the baselines cannot be detected. In this paper, a method is proposed to determine the baselines and relative locations of these invisible cameras. The method consists of two processes executed simultaneously: (a) to statistically detect the baselines among the cameras, and (b) to localize the cameras by using information from (a) and propagating triangle constraints. Process (b) works for the localization in the case where the cameras are observed each other, and it does not require complete observation among the cameras. However, if many cameras cannot be observed each other because of the poor image resolution, it dose not work. The baseline detection by process (a) solves the problem. This methodology is described in detail and results are provided for several scenarios.     

Sparsity Driven People Localization with a Heterogeneous Network of Cameras

This paper addresses the problem of localizing people in low and high density crowds with a network of heterogeneous cameras. The problem is recast as a linear inverse problem. It relies on deducing the discretized occupancy vector of people on the ground, from the noisy binary silhouettes observed as foreground pixels in each camera. This inverse problem is regularized by imposing a sparse occupancy vector, i.e., made of few non-zero elements, while a particular dictionary of silhouettes linearly maps these non-empty grid locations to the multiple silhouettes viewed by the cameras network. The proposed framework is (i) generic to any scene of people, i.e., people are located in low and high density crowds, (ii) scalable to any number of cameras and already working with a single camera, (iii) unconstrained by the scene surface to be monitored, and (iv) versatile with respect to the camera??s geometry, e.g., planar or omnidirectional. Qualitative and quantitative results are presented on the APIDIS and the PETS 2009 Benchmark datasets. The proposed algorithm successfully detects people occluding each other given severely degraded extracted features, while outperforming state-of-the-art people localization techniques.     

Efficient integration of RSSI for tracking using Wireless Camera Networks

This paper proposes a scheme that efficiently exploits synergies between RSSI and camera measurements in cluster-based target tracking using Wireless Camera Networks (WCNs). The scheme is based on the combination of two main components: a training method that accurately trains RSSI-range models adapted to the conditions of the particular local environment; and a sensor activation/deactivation method that decides on the individual activation of sensors balancing the different information contributions and energy consumptions of camera and RSSI measurements involved in sensing. The scheme also includes a distributed Extended Information Filter that integrates all available measurements. The combination of these components originates self-regulated behaviors that drastically reduce power consumption and computational effort with no significant tracking degradation w.r.t. existing schemes based exclusively on cameras. Besides, it shows better robustness to target occlusions. The proposed scheme has been implemented and validated in real experiments.