Towards 3-D scene reconstruction from broadcast video

Three-dimensional (3-D) scene reconstruction from broadcast video is a challenging problem with many potential applications, such as 3-D TV, free-view TV, augmented reality or three-dimensionalization of two-dimensional (2-D) media archives. In this paper, a flexible and effective system capable of efficiently reconstructing 3-D scenes from broadcast video is proposed, with the assumption that there is relative motion between camera and scene/objects. The system requires no a priori information and input, other than the video sequence itself, and capable of estimating the internal and external camera parameters and performing a 3-D motion-based segmentation, as well as computing a dense depth field. The system also serves as a showcase to present some novel approaches for moving object segmentation, sparse and dense reconstruction problems. According to the simulations for both synthetic and real data, the system achieves a promising performance for typical TV content, indicating that it is a significant step towards the 3-D reconstruction of scenes from broadcast video.     

Three-dimensional surface reconstruction using optical flow formedical imaging

The recovery of a three-dimensional (3-D) model from a sequence of two-dimensional (2-D) images is very useful in medical image analysis. Image sequences obtained from the relative motion between the object and the camera or the scanner contain more 3-D information than a single image. Methods to visualize the computed tomograms can be divided into two approaches: the surface rendering approach and the volume rendering approach. In this paper, a new surface rendering method using optical flow is proposed. Optical flow is the apparent motion in the image plane produced by the projection of real 3-D motion onto the 2-D image. The 3-D motion of an object can be recovered from the optical-flow field using additional constraints. By extracting the surface information from 3-D motion, it is possible to obtain an accurate 3-D model of the object. Both synthetic and real image sequences have been used to illustrate the feasibility of the proposed method. The experimental results suggest that the proposed method is suitable for the reconstruction of 3-D models from ultrasound medical images as well as other computed tomograms     

Planar mirrors for image-based robot localization and 3-D reconstruction

Planar catadioptric vision sensors consist of a pinhole camera observing a scene being reflected on two (or more) planar mirrors. These systems have recently received an increasing attention because, unlike stereo cameras, can capture two views of the same scene without the need of hardware multi-camera synchronization and calibration. In this paper we explore the original scenario in which a robot manipulator, equipped with a pinhole camera on its end-effector, observes an unknown 3-D scene both directly and reflected through multiple mirrors. We present new multiple-view properties for this scenario and, based on these theoretical results, we present new image-based camera localization and new 3-D scene reconstruction algorithms. Extensive simulation and real-data experiments illustrate the theory and show the effectiveness of the proposed designs.     

Statistical bias in 3-D reconstruction from a monocular video.

The present state-of-the-art in computing the error statistics in three-dimensional (3-D) reconstruction from video concentrates on estimating the error covariance. A different source of error which has not received much attention is the fact that the reconstruction estimates are often significantly statistically biased. In this paper, we derive a precise expression for the bias in the depth estimate, based on the continuous (differentiable) version of structure from motion (SfM). Many SfM algorithms, or certain portions of them, can be posed in a linear least-squares (LS) framework Ax = b. Examples include initialization procedures for bundle adjustment or algorithms that alternately estimate depth and camera motion. It is a well-known fact that the LS estimate is biased if the system matrix A is noisy. In SfM, the matrix A contains point correspondences, which are always difficult to obtain precisely; thus, it is expected that the structure and motion estimates in such a formulation of the problem would be biased. Existing results on the minimum achievable variance of the SfM estimator are extended by deriving a generalized Cramer-Rao lower bound. A detailed analysis of the effect of various camera motion parameters on the bias is presented. We conclude by presenting the effect of bias compensation on reconstructing 3-D face models from rendered images.     

Camera constraint-free view-based 3-D object retrieval

Recently, extensive research efforts have been dedicated to view-based methods for 3-D object retrieval due to the highly discriminative property of multiviews for 3-D object representation. However, most of state-of-the-art approaches highly depend on their own camera array settings for capturing views of 3-D objects. In order to move toward a general framework for 3-D object retrieval without the limitation of camera array restriction, a camera constraint-free view-based (CCFV) 3-D object retrieval algorithm is proposed in this paper. In this framework, each object is represented by a free set of views, which means that these views can be captured from any direction without camera constraint. For each query object, we first cluster all query views to generate the view clusters, which are then used to build the query models. For a more accurate 3-D object comparison, a positive matching model and a negative matching model are individually trained using positive and negative matched samples, respectively. The CCFV model is generated on the basis of the query Gaussian models by combining the positive matching model and the negative matching model. The CCFV removes the constraint of static camera array settings for view capturing and can be applied to any view-based 3-D object database. We conduct experiments on the National Taiwan University 3-D model database and the ETH 3-D object database. Experimental results show that the proposed scheme can achieve better performance than state-of-the-art methods.     

Nonrigid 2-D/3-D registration for patient specific bronchoscopy simulation with statistical shape modeling: phantom validation

This paper presents a nonrigid registration two-dimensional/three-dimensional (2-D/3-D) framework and its phantom validation for subject-specific bronchoscope simulation. The method exploits the recent development of five degrees-of-freedom miniaturized catheter tip electromagnetic trackers such that the position and orientation of the bronchoscope can be accurately determined. This allows the effective recovery of unknown camera rotation and airway deformation, which is modelled by an active shape model (ASM). ASM captures the intrinsic variability of the tracheo-bronchial tree during breathing and it is specific to the class of motion it represents. The method reduces the number of parameters that control the deformation, and thus greatly simplifies the optimisation procedure. Subsequently, pq-based registration is performed to recover both the camera pose and parameters of the ASM. Detailed assessment of the algorithm is performed on a deformable airway phantom, with the ground truth data being provided by an additional six degrees-of-freedom electromagnetic (EM) tracker to monitor the level of simulated respiratory motion.     

Tracking a dynamic set of feature points

We address the problems of tracking a set of feature points over a long sequence of monocular images as well as how to include and track new feature points detected in successive frames. Due to the 3-D movement of the camera, different parts of the images exhibit different image motion. Tracking discrete features can therefore be decomposed into several independent and local problems. Accordingly, we propose a localized feature tracking algorithm. The trajectory of each feature point is described by a 2-D kinematic model. Then to track a feature point, an interframe motion estimation scheme is designed to obtain the estimates of interframe motion parameters. Subsequently, using the estimates of motion parameters, corresponding points are identified to subpixel accuracy. Afterwards, the temporal information is processed to facilitate the tracking scheme. Since different feature points are tracked independently, the algorithm is able to handle the image motion arising from general 3-D camera movements. On the other hand, in addition to tracking feature points detected at the beginning, an efficient way to dynamically include new points extracted in subsequent frames is devised so that the information in a sequence is preserved. Experimental results for several image sequences are also reported.     

Fine‐Motion Estimation Using Ego/Exo‐Cameras

Robust motion estimation for human–computer interactions played an important role in a novel method of interaction with electronic devices. Existing pose estimation using a monocular camera employs either ego‐motion or exo‐motion, both of which are not sufficiently accurate for estimating fine motion due to the motion ambiguity of rotation and translation. This paper presents a hybrid vision‐based pose estimation method for fine‐motion estimation that is specifically capable of extracting human body motion accurately. The method uses an ego‐camera attached to a point of interest and exo‐cameras located in the immediate surroundings of the point of interest. The exo‐cameras can easily track the exact position of the point of interest by triangulation. Once the position is given, the ego‐camera can accurately obtain the point of interest's orientation. In this way, any ambiguity between rotation and translation is eliminated and the exact motion of a target point (that is, ego‐camera) can then be obtained. The proposed method is expected to provide a practical solution for robustly estimating fine motion in a non‐contact manner, such as in interactive games that are designed for special purposes (for example, remote rehabilitation care systems).     

Real-Time Noncoherent UWB Positioning Radar With Millimeter Range Accuracy: Theory and Experiment

In this paper, we propose a novel architecture for ultra-wideband (UWB) positioning systems, which combines the architectures of carrier-based UWB systems and traditional energy detection-based UWB systems. By implementing the novel architecture, we have successfully developed a standalone noncoherent system for positioning both static and dynamic targets in an indoor environment with approximately 2 and 5 mm of 3-D accuracy, respectively. The results are considered a great milestone in developing such technology. 1-D and 3-D experiments have been carried out and validated using an optical reference system, which provides better than 0.3-mm 3-D accuracy. This type of indoor high-accuracy wireless localization system has many unique applications including robot control, surgical navigation, sensitive material monitoring, and asset tracking.      

Computation and visualization of three-dimensional soft tissue motion in the orbit

This work presents a method to measure the soft tissue motion in three dimensions in the orbit during gaze. It has been shown that two-dimensional (2-D) quantification of soft tissue motion in the orbit is effective in the study of orbital anatomy and motion disorders. However, soft tissue motion is a three-dimensional (3-D) phenomenon and part of the kinematics is lost in any 2-D measurement. Therefore, T1-weighted magnetic resonance (MR) imaging volume sequences are acquired during gaze and soft tissue motion is quantified using a generalization of the Lucas and Kanade optical flow algorithm to three dimensions. New techniques have been developed for visualizing the 3-D flow field as a series of color-texture mapped 2-D slices or as a combination of volume rendering for display of the anatomy and scintillation rendering for the display of the motion field. We have studied the performance of the algorithm on four-dimensional volume sequences of synthetic motion, simulated motion of a static object imaged by MR, an MR-imaged rotating object and MR-imaged motion in the human orbit during gaze. The accuracy of the analysis is sufficient to characterize motion in the orbit and scintillation rendering is an effective visualization technique for 3-D motion in the orbit.     

Ray phase space approach for 3-D imaging and 3-D optical data representation

We propose a framework to model, analyze and design three-dimensional (3-D) imaging systems. A system engineering approach is adopted which relates 3-D images (real or synthesized) to 3-D objects (real or synthesized) using a novel representation of the optical data which we call "ray phase space". The framework provides a powerful tool for determining the performance of 3-D imaging systems, for generating computational reconstruction of 3-D images and for optimizing 3-D imaging systems.     

Application of a 3-DOF parallel manipulator for earthquake simulations

In this paper a formulation and experimental results are presented for a novel application of a 3-degree of freedom (DOF) parallel manipulator to simulate point seismograms and three-dimensional (3-D) earthquake motion. The rigid body acceleration is analyzed to simulate real 3-D earthquakes. Furthermore, first experimental results are reported to analyze earthquake effects on scaled civil structures.     

机载平台6-D运动误差对LiDAR点云质量的影响比较

直升机载荷平台6-D（Six-Dimensional）运动误差（即飞行轨迹和姿态角运动误差）对机载LiDAR点云质量影响显著,进而影响三维重建模型精度。分析各运动误差对点云质量的影响特点,对于有针对性地消除各运动误差影响、有效提高机载LiDAR三维成像产品精度具有重要意义。建立了机载激光扫描脚点三维空间位置偏差与机载平台六方位运动误差之间的传递关系;采用数值仿真,定量比较了六方位运动误差对激光点云密度分布和的影响,获得了六方位运动误差的影响特点及规律。仿真结果表明,直升机载荷平台的三个姿态角运动误差对点云密度的影响更显著,且随飞行高度的增大而增大,而三个飞行轨迹运动误差的影响相对较小。     

3-D model-based frame interpolation for distributed video coding of static scenes.

This paper addresses the problem of side information extraction for distributed coding of videos captured by a camera moving in a 3-D static environment. Examples of targeted applications are augmented reality, remote-controlled robots operating in hazardous environments, or remote exploration by drones. It explores the benefits of the structure-from-motion paradigm for distributed coding of this type of video content. Two interpolation methods constrained by the scene geometry, based either on block matching along epipolar lines or on 3-D mesh fitting, are first developed. These techniques are based on a robust algorithm for sub-pel matching of feature points, which leads to semi-dense correspondences between key frames. However, their rate-distortion (RD) performances are limited by misalignments between the side information and the actual Wyner-Ziv (WZ) frames due to the assumption of linear motion between key frames. To cope with this problem, two feature point tracking techniques are introduced, which recover the camera parameters of the WZ frames. A first technique, in which the frames remain encoded separately, performs tracking at the decoder and leads to significant RD performance gains. A second technique further improves the RD performances by allowing a limited tracking at the encoder. As an additional benefit, statistics on tracks allow the encoder to adapt the key frame frequency to the video motion content.     

Knee surgery assistance: patient model construction, motion simulation, and biomechanical visualization

We present a new system that integrates computer graphics, physics-based modeling, and interactive visualization to assist knee study and surgical operation. First, we discuss generating patient-specific three-dimensional (3-D) knee models from patient's magnetic resonant images (MRIs). The 3-D model is obtained by deforming a reference model to match the MRI dataset. Second, we present simulating knee motion that visualizes patient-specific motion data on the patient-specific knee model. Third, we introduce visualizing biomechanical information on a patient-specific model. The focus is on visualizing contact area, contact forces, and menisci deformation. Traditional methods have difficulty in visualizing knee contact area without using invasive methods. The approach presented here provides an alternative of visualizing the knee contact area and forces without any risk to the patient. Finally, a virtual surgery can be performed. The constructed 3-D knee model is the basis of motion simulation, biomechanical visualization, and virtual surgery. Knee motion simulation determines the knee rotation angles as well as knee contact points. These parameters are used to solve the biomechanical model. Our results integrate 3-D construction, motion simulation, and biomechanical visualization into one system. Overall, the methodologies here are useful elements for future virtual medical systems where all the components of visualization, automated model generation, and surgery simulation come together.     

A novel temporal calibration method for 3-D ultrasound

This paper examines a novel approach for temporal calibration of a three-dimensional (3-D) freehand ultrasound system. A localization system fixed on the probe gives the position and orientation of the probe. For quantitative use, calibration is needed to correctly localize a B-scan in four-dimensional (4-D) (3-D+t) space. Temporal latency estimation is defined in a general robust formulation using no specific probe motion constraints. Experiments were performed on synthetic and real data using a 3-D freehand ultrasound system. The achieved precision is lower than the image acquisition rate (40 ms). A validation study using a calibration phantom has been performed to evaluate the influence of incorrect latency estimation on the 3-D reconstruction procedure. We showed that for latency estimation errors less than 40 ms, the 3-D reconstruction errors are negligible for volume estimation.     

Reconstruction of coronary arteries from a single rotational X-ray projection sequence

Cardiovascular diseases remain the primary cause of death in developed countries. In most cases, exploration of possibly underlying coronary artery pathologies is performed using X-ray coronary angiography. Current clinical routine in coronary angiography is directly conducted in two-dimensional projection images from several static viewing angles. However, for diagnosis and treatment purposes, coronary artery reconstruction is highly suitable. The purpose of this study is to provide physicians with a three-dimensional (3-D) model of coronary arteries, e.g., for absolute 3-D measures for lesion assessment, instead of direct projective measures deduced from the images, which are highly dependent on the viewing angle. In this paper, we propose a novel method to reconstruct coronary arteries from one single rotational X-ray projection sequence. As a side result, we also obtain an estimation of the coronary artery motion. Our method consists of three main consecutive steps: 1) 3-D reconstruction of coronary artery centerlines, including respiratory motion compensation; 2) coronary artery four-dimensional motion computation; 3) 3-D tomographic reconstruction of coronary arteries, involving compensation for respiratory and cardiac motions. We present some experiments on clinical datasets, and the feasibility of a true 3-D Quantitative Coronary Analysis is demonstrated.     

Reconstruction of sculpture from its profiles with unknown camera positions

Profiles of a sculpture provide rich information about its geometry, and can be used for shape recovery under known camera motion. By exploiting correspondences induced by epipolar tangents on the profiles, a successful solution to motion estimation from profiles has been developed in the special case of circular motion. The main drawbacks of using circular motion alone, namely the difficulty in adding new views and part of the object always being invisible, can be overcome by incorporating arbitrary general views of the object and registering its new profiles with the set of profiles resulted from the circular motion. In this paper, we describe a complete and practical system for producing a three-dimensional (3-D) model from uncalibrated images of an arbitrary object using its profiles alone. Experimental results on various objects are presented, demonstrating the quality of the reconstructions using the estimated motion.