Automatic camera control in virtual environments augmented using multiple sparse videos

Automated virtual camera control has been widely used in animation and interactive virtual environments. We have developed a multiple sparse camera based free view video system prototype that allows users to control the position and orientation of a virtual camera, enabling the observation of a real scene in three dimensions (3D) from any desired viewpoint. Automatic camera control can be activated to follow selected objects by the user. Our method combines a simple geometric model of the scene composed of planes (virtual environment), augmented with visual information from the cameras and pre-computed tracking information of moving targets to generate novel perspective corrected 3D views of the virtual camera and moving objects. To achieve real-time rendering performance, view-dependent textured mapped billboards are used to render the moving objects at their correct locations and foreground masks are used to remove the moving objects from the projected video streams. The current prototype runs on a PC with a common graphics card and can generate virtual 2D views from three cameras of resolution 768×576 with several moving objects at about 11 fps.     

Geometric camera calibration using circular control points

Modern CCD cameras are usually capable of a spatial accuracy greater than 1/50 of the pixel size. However, such accuracy is not easily attained due to various error sources that can affect the image formation process. Current calibration methods typically assume that the observations are unbiased, the only error is the zero-mean independent and identically distributed random noise in the observed image coordinates, and the camera model completely explains the mapping between the 3D coordinates and the image coordinates. In general, these conditions are not met, causing the calibration results to be less accurate than expected. In the paper, a calibration procedure for precise 3D computer vision applications is described. It introduces bias correction for circular control points and a nonrecursive method for reversing the distortion model. The accuracy analysis is presented and the error sources that can reduce the theoretical accuracy are discussed. The tests with synthetic images indicate improvements in the calibration results in limited error conditions. In real images, the suppression of external error sources becomes a prerequisite for successful calibration.     

Surveillance camera scheduling: a virtual vision approach

We present a surveillance system, comprising wide field-of-view (FOV) passive cameras and pan/tilt/zoom (PTZ) active cameras, which automatically captures high-resolution videos of pedestrians as they move through a designated area. A wide-FOV static camera can track multiple pedestrians, while any PTZ active camera can capture high-quality videos of one pedestrian at a time. We formulate the multi-camera control strategy as an online scheduling problem and propose a solution that combines the information gathered by the wide-FOV cameras with weighted round-robin scheduling to guide the available PTZ cameras, such that each pedestrian is observed by at least one PTZ camera while in the designated area. A centerpiece of our work is the development and testing of experimental surveillance systems within a visually and behaviorally realistic virtual environment simulator. The simulator is valuable as our research would be more or less infeasible in the real world given the impediments to deploying and experimenting with appropriately complex camera sensor networks in large public spaces. In particular, we demonstrate our surveillance system in a virtual train station environment populated by autonomous, lifelike virtual pedestrians, wherein easily reconfigurable virtual cameras generate synthetic video feeds. The video streams emulate those generated by real surveillance cameras monitoring richly populated public spaces.A preliminary version of this paper appeared as [1].     

Geometric camery calibration: the virtual camera approach

Accurate dimensional measurements made with TV cameras require a careful geometric calibration of the imaging device. This work presents a new calibration approach that makes use of the virtual camera paradigm, in which a calibrated surface (target) plays the role of the image plane of a distortionless, fictitious camera. The method yields a simple closed form solution for the calibration parameters. We use an error analysis to estimate the influence of the calibration set-up on the angular measures. We use simulation to verify the analysis. The results of experimental tests are also reported, indicating good accuracy of the proposed procedure compared to some of the best-known calibration methods.     

Region of interest extraction and virtual camera control based on panoramic video capturing

We present a system for automatically extracting the region of interest (ROI) and controlling virtual cameras' control based on panoramic video. It targets applications such as classroom lectures and video conferencing. For capturing panoramic video, we use the FlyCam system that produces high resolution, wide-angle video by stitching video images from multiple stationary cameras. To generate conventional video, a region of interest can be cropped from the panoramic video. We propose methods for ROI detection, tracking, and virtual camera control that work in both the uncompressed and compressed domains. The ROI is located from motion and color information in the uncompressed domain and macroblock information in the compressed domain, and tracked using a Kalman filter. This results in virtual camera control that simulates human controlled video recording. The system has no physical camera motion and the virtual camera parameters are readily available for video indexing.     

Vision-assisted control for manipulation using virtual fixtures

We present the design and implementation of a vision-based system for cooperative manipulation at millimeter to micrometer scales. The system is based on an admittance control algorithm that implements a broad class of guidance modes called virtual fixtures. A virtual fixture, like a real fixture, limits the motion of a tool to a prescribed class or range of motions. We describe how both hard (unyielding) and soft (yielding) virtual fixtures can be implemented in this control framework. We then detail the construction of virtual fixtures for point positioning and curve following as well as extensions of these to tubes, cones, and sequences thereof. We also describe an implemented system using the JHU Steady Hand Robot. The system uses computer vision as a sensor for providing a reference trajectory, and the virtual fixture control algorithm then provides haptic feedback to implemented direct, shared manipulation. We provide extensive experimental results detailing both system performance and the effects of virtual fixtures on human speed and accuracy.     

Nonlinear active noise control for headrest using virtual microphone control

In this paper, a feed-forward nonlinear active noise control algorithm is developed using the Filtered-S LMS (FSLMS) algorithm and virtual microphone control for an active headrest application. Virtual microphone control is implemented to attenuate noise at the ears of the headrest occupant using the physical microphones placed at remote locations. The proposed nonlinear virtual microphone control algorithm is verified in real-time in a headrest. Different nonlinear noise processes are studied and it is shown that in all cases, the FSLMS algorithm outperforms the FXLMS algorithm in controlling nonlinear primary noise at a virtual location.     

Controlling a camera in a virtual environment

image-based control or visual servoing approach. It consists of positioning a camera according to the information perceived in the image. This is thus a very intuitive approach of animation. To be able to react automatically to modifications of the environment, we also considered the introduction of constraints into the control. This approach is thus adapted to highly reactive contexts (virtual reality, video games). Numerous examples dealing with classic problems in animation are considered within this framework and presented in this paper. Published online: 19 July 2001     

Image feedback path tracking control using an uncalibrated CCD camera

Abstract. Image feedback path tracking (IFPT) control of a laser light point (LLP) using a CCD camera is studied in this paper. The tracking path and the LLP are assumed clearly focused in the scene, but no camera calibration is needed. A modified version of the thinning algorithm SPTA is proposed to skeletonize the path in a piecewise manner. The proposed thinning algorithm takes less computer time than the original SPTA and makes the real-time skeletonization possible. Included in the paper is also the development of a control algorithm with image feedback to assure LLP tracking along the required path, as well as an experimental study to demonstrate how IFPT control can be realized in practice. Received: 10 November 1999 / Accepted: 9 March 2000     

基于视频的静止虚拟摄像机生成

虚拟摄像机的生成是虚拟广告系统中一项关键技术,提出一种静止虚拟摄像机生成方法。首先,在视频图像中自动检测比赛场地的特征点,然后根据特征点在图像坐标系与世界坐标系上的匹配关系,利用摄像机定标技术计算出真实摄像机的内外参数,接着把摄像机的内参数和外参数分别转换为虚拟摄像机的视点转换矩阵和投影矩阵,最后在此虚拟摄像机下对虚拟场景进行渲染,把渲染出的图像插入到视频中。实验结果表明该方法简单、有效、易实现。     

Enhanced teleoperation performance using hybrid control and virtual fixture

To develop secure, natural and effective teleoperation, the perception of the slave plays a key role for the interaction of a human operator with the environment. By sensing slave information, the human operator can choose the correct operation in a process during the human–robot interaction. This paper develops an integrated scheme based on a hybrid control and virtual fixture approach for the telerobot. The human operator can sense the slave interaction condition and adjust the master device via the surface electromyographic signal. This hybrid control method integrates the proportional-derivative control and the variable stiffness control, and involves the muscle activation at the same time. It is proposed to quantitatively analyse the human operator's control demand to enhance the control performance of the teleoperation system. In addition, due to unskilful operation and muscle physiological tremor of the human operator, a virtual fixture method is developed to ensure accuracy of operation and to reduce the operation pressure on the human operator. Experimental results demonstrated the effectiveness of the proposed method for the teleoperated robot.     

Fault-tolerant control of linear systems using adaptive virtual actuator

In this paper, design and development of fault-tolerant control (FTC) is investigated for linear systems subject to loss of effectiveness and time-varying additive actuator faults as well as an external disturbance using the fault-hiding approach. The main aim of this approach is to keep the nominal controller and to design a virtual actuator that is inserted between the faulty plant and the nominal controller in order to hide actuator faults and disturbances from the nominal controller, and consequently the performance of the system before and after the occurrence of actuator faults is kept to be the same. The proposed adaptive virtual actuator does not require a separated fault detection, isolation and identification (FDII) unit and both state and output feedback cases are considered. An illustrative example is given to demonstrate the effectiveness of the proposed adaptive virtual actuator in both cases.     

Inside looking out camera pose estimation for virtual studio

This paper studies the inside looking out camera pose estimation for the virtual studio. The camera pose estimation process, the process of estimating a camera’s extrinsic parameters, is based on closed-form geometrical approaches which use the benefit of simple corner detection of 3D cubic-like virtual studio landmarks. We first look at the effective parameters of the camera pose estimation process for the virtual studio. Our studies include all characteristic landmark parameters like landmark lengths, landmark corner angles and their installation position errors and some camera parameters like lens focal length and CCD resolution. Through computer simulation we investigate and analyze all these parameters’ efficiency in camera extrinsic parameters, including camera rotation and position matrixes. Based on this work, we found that the camera translation vector is affected more than other camera extrinsic parameters because of the noise of effective camera pose estimation parameters. Therefore, we present a novel iterative geometrical noise cancellation method for the closed-form camera pose estimation process. This is based on the collinearity theory that reduces the estimation error of the camera translation vector, which plays a major role in camera extrinsic parameters estimation errors. To validate our method, we test it in a complete virtual studio simulation. Our simulation results show that they are in the same order as those of some commercial systems, such as the BBC and InterSense IS-1200 VisTracker.     

Balance control of a biped robot using camera image of reference object

This paper presents a new balance control scheme for a biped robot. Instead of using dynamic sensors to measure the pose of a biped robot, this paper uses only the visual information of a specific reference object in the workspace. The zero moment point (ZMP) of the biped robot can be calculated from the robot’s pose, which is measured from the reference object image acquired by a CCD camera on the robot’s head. For balance control of the biped robot a servo controller uses an error between the reference ZMP and the current ZMP, estimated by Kalman filter. The efficiency of the proposed algorithm has been proven by the experiments performed on both flat and uneven floors with unknown thin obstacles. Recommended by Editorial Board member Dong Hwan Kim under the direction of Editor Jae-Bok Song. This work was supported by the Korea Research Foundation Grant funded by the Korean Government (MOEHRD). This research was supported by the MKE(The Ministry of Knowledge Economy), Korea, under the ITRC (Information Technology Research Center) support program supervised by the IITA(Institute for Information Technology Advancement) (IITA-2008-C1090-0803-0006). Sangbum Park received the B.S. and M.S. degrees from Electronic Engineering of Soongsil University, Seoul, Korea, in 2004 and 2006 respectively. He has been with School of Electronic Engineering, Soongsil University since 2006, where he is currently pursuing a Ph.D. His current research interests include biped walking robot, robotics vision. Youngjoon Han received the B.S., M.S. and Ph.D. degrees in Electronic Engineering from Soongsil University, Seoul, Korea, in 1996, 1998, and 2003, respectively. He is currently an Assistant Professor in the School of Electornic Engineering at Soongsil University. His research interests include robot vision system, and visual servo control. Hernsoo Hahn received the B.S. and M.S. degrees in Electronic Engineering at Soongsil University and Younsei University, Korea in 1982 and 1983 respectively. He received the Ph.D. degree in Computer Engineering from University of Southern California in 1991, and became an Assistant Professor at the School Electroncis Engneering in Soongsil University in 1992. Currently, he is a Professor. His research interests include application of vision sensors to mobile robots and measurement systems.     

A sensor-based SLAM algorithm for camera tracking in virtual studio

This paper addresses a sensor-based simultaneous localization and mapping （SLAM） algorithm for camera tracking in a virtual studio environment. The traditional camera tracking methods in virtual studios are vision-based or sensor-based. However, the chroma keying process in virtual studios requires color cues, such as blue background, to segment foreground objects to be inserted into images and videos. Chroma keying limits the application of vision-based tracking methods in virtual studios since the background cannot provide enough feature information. Furthermore, the conventional sensor-based tracking approaches suffer from the jitter, drift or expensive computation due to the characteristics of individual sensor system. Therefore, the SLAM techniques from the mobile robot area are first investigated and adapted to the camera tracking area. Then, a sensor-based SLAM extension algorithm for two dimensional （2D） camera tracking in virtual studio is described. Also, a technique called map adjustment is proposed to increase the accuracy＇ and efficiency of the algorithm. The feasibility and robustness of the algorithm is shown by experiments. The simulation results demonstrate that the sensor-based SLAM algorithm can satisfy the fundamental 2D camera tracking requirement in virtual studio environment.     

On active camera control and camera motion recovery with foveatewavelet transform

In this paper, a new variable resolution technique, foveate wavelet transform (FWT), is proposed to represent digital images in an effort to efficiently represent visual data. Compared to existing variable resolution techniques, the strength of the proposed scheme encompasses its linearity preservation, orientation selectivity, and flexibility while supporting interesting behaviors resembling the animate vision system. The linearity preservation of the FWT is due to the fact that only low and/or high-pass filterings are carried out in different regions of an image in the transform. The orientation selectivity indicates the fact that details along the horizontal, vertical, and diagonal directions are readily available in the FWT representation. The flexibility of this new representation technique is witnessed by the readiness of its extensions to represent foveae of different number, shape, and locations. To demonstrate the efficacy of the FWT, two applications are presented. First, an FWT-based active camera control scheme is developed, where the computer can move a camera to track the moving object in the scene. Second, an FWT-based method purporting to recover pan/tilt/zoom camera movements from video clips is developed. Experiments of these two applications have shown encouraging performances     

Extended impedance control using real and virtual sensors for redundant manipulators

Control of a redundant manipulator based on an impedance-control framework with multiple simultaneous control sources is described. Each control source provides a different behavior type. An application is decomposed into multiple simultaneous behaviors whose resultant behavior will provide the motion necessary to execute the task. The simultaneous control inputs are merged using impedance control to compute a resultant command to the manipulator. The task space of each behavior can have the dimensionality of the mechanism being controlled. Control of a seven-degree-of-freedom manipulator is described here with an available task space for each behavior of dimensionality seven.     

Fault tolerant control of uncertain dynamical systems using interval virtual actuators

In this paper, a model reference fault tolerant control strategy based on a reconfiguration of the reference model, with the addition of a virtual actuator block, is presented for uncertain systems affected by disturbances and sensor noise. In particular, this paper (1) extends the reference model approach to the use of interval state observers, by considering an error feedback controller, which uses the estimated bounds for the error between the real state and the reference state, and (2) extends the virtual actuator approach to the use of interval observers, which means that the virtual actuator is added to the control loop to preserve the nonnegativity of the interval estimation errors and the boundedness of the involved signals, in spite of the fault occurrence. In both cases, the conditions to assure the desired operation of the control loop are provided in terms of linear matrix inequalities. An illustrative example is used to show the main characteristics of the proposed approach.     

An integrated advanced process control framework using run-to-run control,virtual metrology and fault detection

This paper develops a new advanced process control (APC) system for the multiple-input multiple-output (MIMO) semiconductor processes using the partial least squares (PLS) technique to provide the run-to-run control with the virtual metrology data, via the gradual mode or the rapid mode depending on the current system status, in order to deal with metrology delays and compensate for different types of system disturbances. First, we present a controller called the PLS-MIMO double exponentially weighted moving average (PLS-MIMO DEWMA) controller. It employs the PLS method as the model building/estimation technique to help the DEWMA controller generate more consistent and robust control outputs than purely using the conventional DEWMA controller. To cope with metrology delays, the proposed APC system uses the pre-processing metrology data to build up the virtual metrology (VM) system that can provide the estimated process outputs for the PLS-MIMO DEWMA controller. Lastly, the Fault Detection (FD) system is added based upon the principal components of the PLS modeling outcomes, which supplies the process status for the VM mechanism and the PLS-MIMO DEWMA controller as to how the process faults are responded. Two scenarios of the simulation study are conducted to illustrate the APC system proposed in this paper.