A high-precision camera operation parameter measurement system andits application to image motion inferring

Information about camera operations such as zoom, focus, pan, tilt and dollying is significant not only for efficient video coding, but also for content-based video representation. In this paper we describe a high-precision camera operation parameter measurement system and apply it to image motion inferring. First, we outline the implemented system which is designed to provide camera operation parameters with a high precision required for image coding applications. Second, we calibrate the camera lens to determine its exact optical properties, A pin-hole camera model with the 2nd order radial lens distortion and a two-image calibration technique are employed. Finally, we use the pan, tilt and zoom parameters measured by the system to infer image motion. The experimental results show that the inferred motion coincides with the actual motion very closely. Compared to the motion analysis techniques that estimate camera motion from video sequences, our approach does not suffer from ambiguity, thus can provide reliable and accurate image global motion. The obtained motion can be applied to image mosaicing, moving object segmentation, object-based image coding, etc     

Dual camera intelligent sensor for high definition 360 degrees surveillance

A novel integrated multi-camera video-sensor (panoramic scene analysis PSA) system is proposed for surveillance applications. In the proposed set-up, an omnidirectional imaging device is used in conjunction with a pan tilt zoom (PTZ) camera leading to an innovative kind of sensor that is able to automatically track at a higher zoom level any moving object within the guarded area. In particular, the catadioptric sensor is calibrated and used in order to track every single moving object within its 360 degree field of view. Omnidirectional image portions are eventually rectified and pan, tilt and zoom parameters of the moving camera are automatically adjusted by the system in order to track detected objects. In addition a co-operative strategy was developed for the selection of the object to be tracked by the PTZ sensor in the case of multiple targets.     

Control Implications in Tracking Moving Objects Using Time-Varying Perspective-Projective Imagery

Control implications which arise when tracking moving objects contained in time-varying perspective-projective imagery are studied. First, a transformation is derived to relate pan/tilt camera mount movement to image plane perturbations. Ramifications of this model, particularly with respect to magnification ratio, noncentered targets and camera mount angular magnitudes are illustrated. A pan/tilt control algorithm is then developed using this transformation and a large magnification ratio assumption. Experimental results with real imagery are shown to confirm the approach validity.     

Collaborative sensing in a distributed PTZ camera network

The performance of dynamic scene algorithms often suffers because of the inability to effectively acquire features on the targets, particularly when they are distributed over a wide field of view. In this paper, we propose an integrated analysis and control framework for a pan, tilt, zoom (PTZ) camera network in order to maximize various scene understanding performance criteria (e.g., tracking accuracy, best shot, and image resolution) through dynamic camera-to-target assignment and efficient feature acquisition. Moreover, we consider the situation where processing is distributed across the network since it is often unrealistic to have all the image data at a central location. In such situations, the cameras, although autonomous, must collaborate among themselves because each camera's PTZ parameter entails constraints on the others. Motivated by recent work in cooperative control of sensor networks, we propose a distributed optimization strategy, which can be modeled as a game involving the cameras and targets. The cameras gain by reducing the error covariance of the tracked targets or through higher resolution feature acquisition, which, however, comes at the risk of losing the dynamic target. Through the optimization of this reward-versus-risk tradeoff, we are able to control the PTZ parameters of the cameras and assign them to targets dynamically. The tracks, upon which the control algorithm is dependent, are obtained through a consensus estimation algorithm whereby cameras can arrive at a consensus on the state of each target through a negotiation strategy. We analyze the performance of this collaborative sensing strategy in active camera networks in a simulation environment, as well as a real-life camera network.     

A helmet mounted display system with active gaze control for visual telepresence

The objective of a Visual Telepresence System is to provide the operator with a high fidelity image from a remote stereo camera pair linked to a pan/tilt device such that the operator may reorient the camera position by use of head movement. Systems such as these which utilise virtual reality style helmet mounted displays have a number of limitations. The geometry of the camera positions and of the displays is generally fixed and is most suitable only for viewing elements of a scene at a particular distance. To address such limitations, a prototype system has been developed where the geometry of the displays and cameras is dynamically controlled by the eye movement of the operator. This paper explores why it is necessary to actively adjust the display system as well as the cameras and justifies the use of mechanical adjustment of the displays as an alternative to adjustment by electronic or image processing methods. The electronic and mechanical design is described including optical arrangements and control algorithms. The performance and accuracy of the system is assessed with respect to eye movement.     

Simultaneous homographic and comparametric alignment of multiple exposure-adjusted pictures of the same scene

An approach is presented that can simultaneously align multiple exposure-adjusted pictures of the same scene both in their spatial coordinates as well as in their pixel values. The approach is featureless and produces an image mosaic at a common spatial and exposure reference and also addresses the misalignment problem common to methods that compose mosaics from only pair-wise registered image pairs. The objective function considered minimizes the sum of the collective variance over pixels of a global coordinate grid on which to create the final image. The models employed relate images spatially by homographic transformations and tonally by comparametric functions. The importance of performing joint spatial and tonal registration on exposure-adjusted images is emphasized by providing two examples in which spatial-only registration fails. The performance between pair-wise and simultaneous registration under both spatial-only and joint registration procedures is discussed.     

An information fidelity criterion for image quality assessment using natural scene statistics.

Measurement of visual quality is of fundamental importance to numerous image and video processing applications. The goal of quality assessment (QA) research is to design algorithms that can automatically assess the quality of images or videos in a perceptually consistent manner. Traditionally, image QA algorithms interpret image quality as fidelity or similarity with a "reference" or "perfecft" image in some perceptual space. Such "full-referenc" QA methods attempt to achieve consistency in quality prediction by modeling salient physiological and psychovisual features of the human visual system (HVS), or by arbitrary signal fidelity criteria. In this paper, we approach the problem of image QA by proposing a novel information fidelity criterion that is based on natural scene statistics. QA systems are invariably involved with judging the visual quality of images and videos that are meant for "human consumption." Researchers have developed sophisticated models to capture the statistics of natural signals, that is, pictures and videos of the visual environment. Using these statistical models in an information-theoretic setting, we derive a novel QA algorithm that provides clear advantages over the traditional approaches. In particular, it is parameterless and outperforms current methods in our testing. We validate the performance of our algorithm with an extensive subjective study involving 779 images. We also show that, although our approach distinctly departs from traditional HVS-based methods, it is functionally similar to them under certain conditions, yet it outperforms them due to improved modeling. The code and the data from the subjective study are available at.     

一种基于参数重估计和分层匹配的PTZ相机参数修正算法

由于多种因素的影响,PTZ(pan-ti lt-zoom)相机经过长时间运行后其参数会偏离真实值,因 此有必要对相机参数进行修正。针对现有算法参数修正精度低、适应能力弱等问题,提 出了一 种改进的基于参数重估计和分层匹配的参数修正算法。算法通过引入参数重估计策略,避免 了参数 修正过程中的误差累积,提高了参数修正的精度;通过设计分层匹配和特征传播步骤,增强 了算法 对不同尺度图像的适应性。实际场景中的多组实验结果表明,本文算法可以准确修正PTZ相 机参数,并且较现有方法更具优越性。     

Determining the absolute orientation in a corridor using projectivegeometry and active vision

The capability of a mobile robot to determine its position in the environment (self-localization) is a prerequisite for achieving autonomous navigation. An approach is proposed for determining the absolute orientation of an autonomous robot in a system of corridors, based on the projective geometry and active computer vision. In the proposed approach, the common direction of longitudinal corridor edges is inferred by detecting the vanishing point of the corresponding straight line segments in the image. It is assumed that the knowledge about the vertical direction in the scene is available, so that the image coordinates of these vanishing points are considerably constrained. However, longitudinal corridor edges are not visible in images acquired for many viewing directions, so that the processing in a localization procedure has to be performed on a sequence of images acquired from the given position, for regularly arranged orientations of the camera. Extensive experimentation was performed on real scenes and the obtained results are provided     

Three-dimensional reconstruction using homocentric spherical spatiotemporal image analysis

This paper presents a new algorithm for reconstructing a scene of three-dimensional structures from an image sequence. Three-dimensional reconstruction using an image sequence, called the spatiotemporal image method, is robust against image noises. But in this method, camera motion is limited to only one direction translation. Our algorithm makes allowances for camera rotation in spatiotemporal image analysis. With this technique, the whole spatiotemporal image is transformed to spherical projection and three-dimensional structures are determined robustly using the Hough transformation. We call the technique Homocentric Spherical Spatiotemporal Image (HSSI) analysis. With HSSI, it is possible to distinguish objects with a rotating camera from a longer baseline and to measure them with much greater accuracy than previously possible. This algorithm is demonstrated through simulations and experiments with real images from a translating and rotating camera, and the three-dimensional structures in a static scene are reconstructed.     

Super-resolution image reconstruction: a technical overview

A new approach toward increasing spatial resolution is required to overcome the limitations of the sensors and optics manufacturing technology. One promising approach is to use signal processing techniques to obtain an high-resolution (HR) image (or sequence) from observed multiple low-resolution (LR) images. Such a resolution enhancement approach has been one of the most active research areas, and it is called super resolution (SR) (or HR) image reconstruction or simply resolution enhancement. In this article, we use the term "SR image reconstruction" to refer to a signal processing approach toward resolution enhancement because the term "super" in "super resolution" represents very well the characteristics of the technique overcoming the inherent resolution limitation of LR imaging systems. The major advantage of the signal processing approach is that it may cost less and the existing LR imaging systems can be still utilized. The SR image reconstruction is proved to be useful in many practical cases where multiple frames of the same scene can be obtained, including medical imaging, satellite imaging, and video applications. The goal of this article is to introduce the concept of SR algorithms to readers who are unfamiliar with this area and to provide a review for experts. To this purpose, we present the technical review of various existing SR methodologies which are often employed. Before presenting the review of existing SR algorithms, we first model the LR image acquisition process.     

Recognition of dynamic video contents with global probabilistic models of visual motion.

The exploitation of video data requires methods able to extract high-level information from the images. Video summarization, video retrieval, or video surveillance are examples of applications. In this paper, we tackle the challenging problem of recognizing dynamic video contents from low-level motion features. We adopt a statistical approach involving modeling, (supervised) learning, and classification issues. Because of the diversity of video content (even for a given class of events), we have to design appropriate models of visual motion and learn them from videos. We have defined original parsimonious global probabilistic motion models, both for the dominant image motion (assumed to be due to the camera motion) and the residual image motion (related to scene motion). Motion measurements include affine motion models to capture the camera motion and low-level local motion features to account for scene motion. Motion learning and recognition are solved using maximum likelihood criteria. To validate the interest of the proposed motion modeling and recognition framework, we report dynamic content recognition results on sports videos.     

采用双目立体相机的实时集成成像拍摄系统

提出了一种基于双目立体相机的实时集成成像拍摄系统。不同于采用传统的摄像机阵列,该系统采用双目相机对三维场景进行拍摄,有效地简化了集成成像拍摄系统的结构。该系统首先利用双目相机获取三维场景的左右视差图,然后上传到图形处理器生成三维场景的高分辨率深度图,之后利用深度图和彩色纹理图在图形处理器中并行生成新视点视差图像,并利用像素映射算法生成高分辨率微图像阵列,实现实时的集成成像显示。实验中系统获取的深度图像素数目是微软Kinect2获取深度图像素数目的4.25倍,当系统运行在1 920 pixel1 080 pixel、99视点数的环境下,可实现三维场景的实时拍摄与显示,实验结果证明了所提系统的可行性。     

New methods for dynamic mosaicking

This paper presents a new technique for the creation of a sequence of mosaic images from an original video shot. A mosaic image represents, on a single image, the scene background seen all over the sequence and its creation requires the estimation of the warping parameters and the use of a blending technique. The warping parameters permit one to represent each original image in the mosaic reference. An estimation method, based on a direct comparison between the current original image and the previously calculated mosaic is proposed. A new analytic minimization criterion is also designed to optimize the determination of the blending coefficient used for the update of the mosaic image with a new original image. This criterion is based on constraints related to the temporal variations of the background, the temporal delay and the resolution of the created mosaic images, while its minimization can be analytically performed. Finally, the proposed method is applied to the creation of new video sequences in which the camera point of view, the camera focal, or the image size are modified. This approach has been tested and validated on real video sequences with large camera motion.     

混合式光场相机在聚焦与非聚焦模式下的超分辨重聚焦仿真

光场描述了光在自由空间传播的全四维信息,光场相机可用来获得光场图像。在传统的光场相机中,最终获得图像的空间分辨率受限于微透镜阵列中透镜的个数。聚焦型光场相机相较于传统光场相机能够获得更高的空间分辨率,但是以牺牲其角度分辨率作为代价。在Zemax中建立了传统光场相机与将聚焦光场相机的成像模型,仿真获得了两种光场相机的光场图像,分析了两种不同类型光场相机采样模式的区别。提出将可变焦液体透镜阵列放置在光场相机中,可以同时获得聚焦和非聚焦两种模式下的光场图像。根据记录的光场信息,讨论了相应的重聚焦方法,计算仿真了在不同景深下的重聚焦图像,并提出了一种基于图像融合和超分辨率重构的方法来提高重聚焦图像的分辨率,最终在相同的景深范围内获得了3倍于传统光场相机分辨率的重聚焦图像。     

基于PCA的相机响应函数模型标定算法

许多计算机视觉应用的算法都需要对拍摄场景高动态范围的幅亮度信息进行精确的测量,成像系统的相机响应函数能够建立拍摄图像强度信息与场景辐亮度之间的严格映射关系,是高动态范围图像融合的关键技术。文中分析相机响应曲线的共同特点,结合相机响应函数固有的约束条件,建立相机响应函数的理论空间模型。首先,利用主成分分析法对已有的相机响应数据库进行分析,结合相机响应函数的约束条件建立响应函数的低参数经验模型;然后,根据输入图像选择合适的参数数量;最后,利用不同曝光量的输入图像通过最小二乘法求解建立响应函数模型的系数,从而对相机响应函数进行标定。该算法能够通过对少量的采样点进行插值获得精确的相机响应函数,同时能够对任意的场景通过拍摄多曝光量图像精确地标定相机响应函数。通过对实际拍摄的图像进行相机响应函数标定实验,验证了该算法的有效性,并证明该算法保持高精度的同时计算效率也较高。     

虚拟演播室中节目主持人或演员与虚拟场景融合的一种技术方案

研制了用跟踪器和测距器改进传统BETACAM摄像机并与主机软件模块链接的虚拟演播室系统，提出了相机-后相机偶合对模型以解决节目主持人对虚拟场景的融入问题并与实际BETACAM摄像机进行捆绑，导出了以实拍BETACAM摄像机投影中心点深度信息为区分界限的虚拟场景前景帧缓存和背景帧缓存的成像公式，达到了实拍节目主持人视频图像与虚拟场景的计算机CG图像在空间位置及其透视投影关系、图像对象掩模关系上的实时一致的效果，实验结果表明文中所研制的虚拟演播室构造技术是可行的，比目前已有的基于抠像及与背景合成的视频编辑技术的虚拟演播室技术更为理想和实用。     

从单幅图像恢复立体景象的新方法

在研究单幅平面图像内在特性的基础上,提出了一种恢复立体视觉景象建模的新方法。对图像进行智能识别处理,可以求得许多线段的特征参数,并由此计算出消隐点和消隐线,从而可自动获得场景的立体结构信息。本算法的特点在于用一个代数表达式统一了三种典型的度量方法,无需传统的相机内校正参数,直接可计算出建模用立体信息。建模结果用VRML格式保存、输出,以便于网上浏览。众多的图像验证了该方法的有效性、适用性。     

Ghost detection and removal for high dynamic range images: Recent advances

High dynamic range (HDR) image generation and display technologies are becoming increasingly popular in various applications. A standard and commonly used approach to obtain an HDR image is the multiple exposures' fusion technique which consists of combining multiple images of the same scene with varying exposure times. However, if the scene is not static during the sequence acquisition, moving objects manifest themselves as ghosting artefacts in the final HDR image. Detecting and removing ghosting artefacts is an important issue for automatically generating HDR images of dynamic scenes. The aim of this paper is to provide an up-to-date review of the recently proposed methods for ghost-free HDR image generation. Moreover, a classification and comparison of the reviewed methods is reported to serve as a useful guide for future research on this topic.     

Jointly registering images in domain and range by piecewise linear comparametric analysis

This paper describes an approach whereby comparametric analysis is used in jointly registering image pairs in their domain and range, i.e., in their spatial coordinates and pixel values, respectively. This is accomplished by approximating a camera's nonlinear comparametric function with a constrained piecewise linear one. The optimal fitting of this approximation to comparagram data is then used in a re-parameterized version of the camera's comparametric function to estimate the exposure difference between images. Doing this allows the inherently nonlinear problem of joint domain and range registration to be performed using a computationally attractive least squares formalism. The paper first presents the range registration process and then describes the strategy for performing the joint registration. The models used allow for the pair-wise registration of images taken from a camera that can automatically adjust its exposure as well as tilt, pan, rotate and zoom about its optical center. Results concerning the joint registration as well as range-only registration are provided to demonstrate the method's effectiveness.