Statistical shape model-based femur kinematics from biplane fluoroscopy

Studying joint kinematics is of interest to improve prosthesis design and to characterize postoperative motion. State of the art techniques register bones segmented from prior computed tomography or magnetic resonance scans with X-ray fluoroscopic sequences. Elimination of the prior 3D acquisition could potentially lower costs and radiation dose. Therefore, we propose to substitute the segmented bone surface with a statistical shape model based estimate. A dedicated dynamic reconstruction and tracking algorithm was developed estimating the shape based on all frames, and pose per frame. The algorithm minimizes the difference between the projected bone contour and image edges. To increase robustness, we employ a dynamic prior, image features, and prior knowledge about bone edge appearances. This enables tracking and reconstruction from a single initial pose per sequence. We evaluated our method on the distal femur using eight biplane fluoroscopic drop-landing sequences. The proposed dynamic prior and features increased the convergence rate of the reconstruction from 71% to 91%, using a convergence limit of 3 mm. The achieved root mean square point-to-surface accuracy at the converged frames was 1.48 ± 0.41 mm. The resulting tracking precision was 1-1.5 mm, with the largest errors occurring in the rotation around the femoral shaft (about 2.5° precision).     

Markerless real-time 3-D target region tracking by motion backprojection from projection images

Accurate and fast localization of a predefined target region inside the patient is an important component of many image-guided therapy procedures. This problem is commonly solved by registration of intraoperative 2-D projection images to 3-D preoperative images. If the patient is not fixed during the intervention, the 2-D image acquisition is repeated several times during the procedure, and the registration problem can be cast instead as a 3-D tracking problem. To solve the 3-D problem, we propose in this paper to apply 2-D region tracking to first recover the components of the transformation that are in-plane to the projections. The 2-D motion estimates of all projections are backprojected into 3-D space, where they are then combined into a consistent estimate of the 3-D motion. We compare this method to intensity-based 2-D to 3-D registration and a combination of 2-D motion backprojection followed by a 2-D to 3-D registration stage. Using clinical data with a fiducial marker-based gold-standard transformation, we show that our method is capable of accurately tracking vertebral targets in 3-D from 2-D motion measured in X-ray projection images. Using a standard tracking algorithm (hyperplane tracking), tracking is achieved at video frame rates but fails relatively often (32% of all frames tracked with target registration error (TRE) better than 1.2 mm, 82% of all frames tracked with TRE better than 2.4 mm). With intensity-based 2-D to 2-D image registration using normalized mutual information (NMI) and pattern intensity (PI), accuracy and robustness are substantially improved. NMI tracked 82% of all frames in our data with TRE better than 1.2 mm and 96% of all frames with TRE better than 2.4 mm. This comes at the cost of a reduced frame rate, 1.7 s average processing time per frame and projection device. Results using PI were slightly more accurate, but required on average 5.4 s time per frame. These results are still substantially faster than 2-D to 3-D registration. We conclude that motion backprojection from 2-D motion tracking is an accurate and efficient method for tracking 3-D target motion, but tracking 2-D motion accurately and robustly remains a challenge.     

Multiple camera tracking of interacting and occluded human motion

We propose a distributed, real-time computing platform for tracking multiple interacting persons in motion. To combat the negative effects of occlusion and articulated motion we use a multiview implementation, where each view is first independently processed on a dedicated processor. This monocular processing uses a predictor-corrector filter to weigh reprojections of three-dimensional (3-D) position estimates, obtained by the central processor, against observations of measurable image motion. The corrected state vectors from each view provide input observations to a Bayesian belief network, in the central processor, with a dynamic, multidimensional topology that varies as a function of scene content and feature confidence. The Bayesian net fuses independent observations from multiple cameras by iteratively resolving independency relationships and confidence levels within the graph, thereby producing the most likely vector of 3-D state estimates given the available data. To maintain temporal continuity, we follow the network with a layer of Kalman filtering that updates the 3-D state estimates. We demonstrate the efficacy of the proposed system using a multiview sequence of several people in motion. Our experiments suggest that, when compared with data fusion based on averaging, the proposed technique yields a noticeable improvement in tracking accuracy     

Tracking a dynamic set of feature points

We address the problems of tracking a set of feature points over a long sequence of monocular images as well as how to include and track new feature points detected in successive frames. Due to the 3-D movement of the camera, different parts of the images exhibit different image motion. Tracking discrete features can therefore be decomposed into several independent and local problems. Accordingly, we propose a localized feature tracking algorithm. The trajectory of each feature point is described by a 2-D kinematic model. Then to track a feature point, an interframe motion estimation scheme is designed to obtain the estimates of interframe motion parameters. Subsequently, using the estimates of motion parameters, corresponding points are identified to subpixel accuracy. Afterwards, the temporal information is processed to facilitate the tracking scheme. Since different feature points are tracked independently, the algorithm is able to handle the image motion arising from general 3-D camera movements. On the other hand, in addition to tracking feature points detected at the beginning, an efficient way to dynamically include new points extracted in subsequent frames is devised so that the information in a sequence is preserved. Experimental results for several image sequences are also reported.     

Real-time multimodal retinal image registration for a computer-assisted laser photocoagulation system

An algorithm for the real-time registration of a retinal video sequence captured with a scanning digital ophthalmoscope (SDO) to a retinal composite image is presented. This method is designed for a computer-assisted retinal laser photocoagulation system to compensate for retinal motion and hence enhance the accuracy, speed, and patient safety of retinal laser treatments. The procedure combines intensity and feature-based registration techniques. For the registration of an individual frame, the translational frame-to-frame motion between preceding and current frame is detected by normalized cross correlation. Next, vessel points on the current video frame are identified and an initial transformation estimate is constructed from the calculated translation vector and the quadratic registration matrix of the previous frame. The vessel points are then iteratively matched to the segmented vessel centerline of the composite image to refine the initial transformation and register the video frame to the composite image. Criteria for image quality and algorithm convergence are introduced, which assess the exclusion of single frames from the registration process and enable a loss of tracking signal if necessary. The algorithm was successfully applied to ten different video sequences recorded from patients. It revealed an average accuracy of 2.47 ± 2.0 pixels (～23.2 ± 18.8 μm) for 2764 evaluated video frames and demonstrated that it meets the clinical requirements.     

Contour image sequence compression through motion analysis and hybrid coding method

This paper presents a motion analysis algorithm (MAA) and a hybrid coding method for contour image sequence compression. The contour image sequence consists of objects moving and rotating in a 3-D world with occlusion, shape, and size variations from frame to frame. The MAA separates the moving image sequence into several object-oriented subsequences (OOSs). In each OOS, the component is either stationary or moves smoothly, and the motion parameters can be easily estimated. The first and last frames of OOS are key frames, and the others are in-between frames. The key frames are unpredictable, and the entire frames need to be encoded. The in-between frames are compensable, and they are encoded by the motion parameter coding. The hybrid coder uses vectorgraph coding to remove spatial redundancy of the key frames and motion parameter coding to reduce the temporal redundancy of the OOSs. The motion parameters are encoded as combinations of 2-D translation, 2-D rotation, and scaling. There are many applications for contour image sequence compression. The cartoon image sequence (a sequence of line drawing sketches) and the high-frame-rate videophone for sign language transmission are good examples. Experiments show that our method encodes the contour image sequence at a very high compression ratio without losing intelligibility.     

Robust bilayer video segmentation by adaptive propagation of global shape and local appearance

Segmenting semantic objects of interest from video has long been an active research topic, with a wide range of potential applications. In this paper, we present a bilayer video segmentation method robust to abrupt motion and change in appearance for both the foreground and background. Specifically, based on a few manually segmented keyframes, the proposed method propagates the global shape of the foreground as priors to adjacent frames by applying branch-and-mincut [1], which jointly estimates what is optimal among a set of shapes along with its pose and the corresponding segmentation in the current image. Based on this preliminary segmentation we determine two types of local regions likely to have erroneous results, and apply a probabilistic framework where shape and appearance cues are adaptively emphasized for local refinement. With each successive frame segmentation, the set of shapes applied as priors are incrementally updated. Experimental results support the robustness of the proposed method for obstacles such as background clutter, motion, and appearance changes, from only a small number of user segmented keyframes.     

多视角下结合形状和运动信息的三维人体姿态估计

该文以多视角同步视频为输入,提出综合利用形状和运动信息的3维人体姿态估计方法。该方法将人体分为头、躯干和四肢等3部分,每部分利用运动信息来预测当前的状态,并以形状信息作为检测器来确定姿态。这种在姿态估计中使用互补信息的方式极大地解决了漂移和收敛到局部极小的问题,也使系统能自动初始化和失败后重初始化。同时,多视角数据的使用也解决了自遮挡问题和运动歧义性。在包含多种运动类型的序列上的测试结果说明了该方法的有效性,对比实验结果也优于Condensation算法和退火粒子滤波。     

基于IEKF视觉运动分析递归算法的研究

本文提出基于图象序列上特征点的图象平面轨迹进行运动参数估计，以提取运动物体的姿态、速度以及外部环境内所感兴趣点的位置参数，由于图象特征点轨迹与待估计的参数有关，因此能够组合长序列图象内的信息并利用ＩＥＫＦ递归地估计未知的运动及结构参数。     

针对运动摄像机的快速低存储开销运动目标检测算法

摄像机的运动会导致整幅图像的运动,使得此情形下的目标检测极具挑战性。针对该问题提出一种快速低存储开销检测算法。首先,利用一种快速低存储开销配准方法计算相邻两帧的单应变换矩阵。而后,使用单应变换矩阵进行相邻两帧之间的配准,并由帧间差分获取帧间运动信息。最后,采用积累运动信息的方式构造不断更新的运动图像,通过对此运动图像进行阈值分割分离出最终的运动目标。在多个不同视频序列下的实验表明该算法能够有效地从嘈杂的场景中检测出运动目标。此外,与先前算法相比,该算法检测性能更好,且显著地降低了存储开销与计算时间开销。对于480360的序列而言,该算法需要的存储开销仅为825 kByte,且运算速度达到16帧/m。     

一种基于对象的动态视频超分辨率复原算法

从视频序列中复原高分辨率的运动对象在众多研究领域具有重要的应用意义.本文针对动态视频中整体运动的刚性或准刚性对象,提出一种基于对象的超分辨率复原方案,首先引入基于6参数仿射模型的对象跟踪和匹配算法,用于视频中运动对象的自动跟踪和匹配.进而将该运动模型与最大后验概率(MAP)算法相结合实现了所跟踪对象的超分辨率复原.对仿真和实测序列的实验结果表明,这种基于对象的处理方法能够实现更为准确的运动估计,因而收到了更好的复原效果.     

Exploiting Motion Correlations in 3-D Articulated Human Motion Tracking

In 3-D articulated human motion tracking, the curse of dimensionality renders commonly-used particle-filter-based approaches inefficient. Also, noisy image measurements and imperfect feature extraction call for strong motion prior. We propose to learn the correlation between the right-side and the left-side human motion using partial least square (PLS) regression. The correlation effectively constrains the sampling of the proposal distribution to portions of the parameter space that correspond to plausible human motions. The learned correlation is then used as motion prior in designing a Rao–Blackwellized particle filter algorithm, RBPF-PLS, which estimates only one group of state variables using the Monte Carlo method, leaving the other group being exactly computed through an analytical filter that utilizes the learned motion correlation. We quantitatively assessed the accuracy of the proposed algorithm with challenging HumanEva-I/II data set. Experiments with comparison with both the annealed particle filter and the standard particle filter show that the proposed method achieves lower estimation error in processing challenging real-world data of 3-D human motion. In particular, the experiments demonstrate that the learned motion correlation model generalizes well to motions outside of the training set and is insensitive to the choice of the training subjects, suggesting the potential wide applicability of the method.      

Fine‐Motion Estimation Using Ego/Exo‐Cameras

Robust motion estimation for human–computer interactions played an important role in a novel method of interaction with electronic devices. Existing pose estimation using a monocular camera employs either ego‐motion or exo‐motion, both of which are not sufficiently accurate for estimating fine motion due to the motion ambiguity of rotation and translation. This paper presents a hybrid vision‐based pose estimation method for fine‐motion estimation that is specifically capable of extracting human body motion accurately. The method uses an ego‐camera attached to a point of interest and exo‐cameras located in the immediate surroundings of the point of interest. The exo‐cameras can easily track the exact position of the point of interest by triangulation. Once the position is given, the ego‐camera can accurately obtain the point of interest's orientation. In this way, any ambiguity between rotation and translation is eliminated and the exact motion of a target point (that is, ego‐camera) can then be obtained. The proposed method is expected to provide a practical solution for robustly estimating fine motion in a non‐contact manner, such as in interactive games that are designed for special purposes (for example, remote rehabilitation care systems).     

Nonrigid 2-D/3-D registration for patient specific bronchoscopy simulation with statistical shape modeling: phantom validation

This paper presents a nonrigid registration two-dimensional/three-dimensional (2-D/3-D) framework and its phantom validation for subject-specific bronchoscope simulation. The method exploits the recent development of five degrees-of-freedom miniaturized catheter tip electromagnetic trackers such that the position and orientation of the bronchoscope can be accurately determined. This allows the effective recovery of unknown camera rotation and airway deformation, which is modelled by an active shape model (ASM). ASM captures the intrinsic variability of the tracheo-bronchial tree during breathing and it is specific to the class of motion it represents. The method reduces the number of parameters that control the deformation, and thus greatly simplifies the optimisation procedure. Subsequently, pq-based registration is performed to recover both the camera pose and parameters of the ASM. Detailed assessment of the algorithm is performed on a deformable airway phantom, with the ground truth data being provided by an additional six degrees-of-freedom electromagnetic (EM) tracker to monitor the level of simulated respiratory motion.     

姿态估计和跟踪结合的运动视频关键帧提取

提出姿态估计和特定部位跟踪相结合的动作视频关键帧提取算法.首先利用非确定人体部位的时间连续性保持提高基于柔性部件铰接人体模型的单帧图像人体姿态估计准确率,通过实施数据降维得到局部拓扑结构表达能力强的判别性运动特征向量,采用极值判定原理确定候选关键帧集合.然后利用ISODATA动态聚类算法,通过初始聚类中心优化、基于语义的关键帧集合增强等策略确定关键帧.实验表明文中算法具有较高的关键帧提取准确率和召回率,支持基于语义的关键帧提取.提取的视频关键帧可以用于运动视频压缩和批注审阅.     

Homography-based block motion estimation for video coding of PTZ cameras

Due to the constrained movement of pan-tilt-zoom (PTZ) cameras, two frames in the video sequences captured by such cameras can be geometrically related by a relationship (homography). This geometric relationship is helpful for reducing the spatial redundancy in video coding. In this paper, by exploiting the homography between two frames with optical flow tracking algorithm, we propose a novel homography-based search (HBS) algorithm for block motion estimation in coding the sequences captured by PTZ cameras. In addition, adaptive thresholds are adopted in our method to classify different kinds of blocks. Compared with other traditional fast algorithms, the proposed HBS algorithm is proved to be more efficient for the sequences captured by PTZ cameras. And compared to our previous work in ICME (Cui et al., 2011), which only deals with pan-tilt (PT) camera and calculates the homography with mechanical devices, in this extended work we compute the homography by using information on images instead.     

Space-time image sequence analysis: object tunnels and occlusion volumes.

We address the issue of image sequence analysis jointly in space and time. While typical approaches to such an analysis consider two image frames at a time, we propose to perform this analysis jointly over multiple frames. We concentrate on spatiotemporal segmentation of image sequences and on analysis of occlusion effects therein. The segmentation process is three-dimensional (3-D); we search for a volume carved out by each moving object in the image sequence domain, or "object tunnel," a new space-time concept. We pose the problem in variational framework by using only motion information (no intensity edges). The resulting formulation can be viewed as volume competition, a 3-D generalization of region competition. We parameterize the unknown surface to be estimated, but rather than using an active-surface approach, we embed it into a higher dimensional function and apply the level-set methodology. We first develop simple models for the detection of moving objects over static background; no motion models are needed. Then, in order to improve segmentation accuracy, we incorporate motion models for objects and background. We further extend the method by including explicit models for occluded and newly exposed areas that lead to "occlusion volumes," another new space-time concept. Since, in this case, multiple volumes are sought, we apply a multiphase variant of the level-set method. We present various experimental results for synthetic and natural image sequences.     

一种直线轨迹跟踪的视觉传感器

提出了一种实现直线轨迹跟踪的视觉传感器方案，详细讨论了传感器的基本结构和工作原理，并对其中涉及的数字图像处理技术进行了深入研究。该传感器由激光扫描单元和图像采集处理单元2部分组成，其中，激光扫描单元采用三角法测量原理，利用一维视觉传感扫描的方法实现直线运动物体的轨迹跟踪；图像采集处理单元则利用FPGA和DSP对CCD的输出数据进行高速的数字化处理，并引导后端的执行机构进行相应动作。实验表明，本传感器能以60line／s的速度对被测物进行扫描，并在主机界面上清晰地显示被测物的直线距离信息，测量分辨率可达14μm。     

Constrained registration for motion compensation in atrial fibrillation ablation procedures

Fluoroscopic overlay images rendered from preoperative volumetric data can provide additional anatomical details to guide physicians during catheter ablation procedures for treatment of atrial fibrillation (AFib). As these overlay images are often compromised by cardiac and respiratory motion, motion compensation methods are needed to keep the overlay images in sync with the fluoroscopic images. So far, these approaches have either required simultaneous biplane imaging for 3-D motion compensation, or in case of monoplane X-ray imaging, provided only a limited 2-D functionality. To overcome the downsides of the previously suggested methods, we propose an approach that facilitates a full 3-D motion compensation even if only monoplane X-ray images are available. To this end, we use a training phase that employs a biplane sequence to establish a patient specific motion model. Afterwards, a constrained model-based 2-D/3-D registration method is used to track a circumferential mapping catheter. This device is commonly used for AFib catheter ablation procedures. Based on the experiments on real patient data, we found that our constrained monoplane 2-D/3-D registration outperformed the unconstrained counterpart and yielded an average 2-D tracking error of 0.6 mm and an average 3-D tracking error of 1.6 mm. The unconstrained 2-D/3-D registration technique yielded a similar 2-D performance, but the 3-D tracking error increased to 3.2 mm mostly due to wrongly estimated 3-D motion components in X-ray view direction. Compared to the conventional 2-D monoplane method, the proposed method provides a more seamless workflow by removing the need for catheter model re-initialization otherwise required when the C-arm view orientation changes. In addition, the proposed method can be straightforwardly combined with the previously introduced biplane motion compensation technique to obtain a good trade-off between accuracy and radiation dose reduction.     

Three-dimensional modeling from two-dimensional video

This paper presents the surface-based factorization method to recover three-dimensional (3-D) structure, i.e., the 3-D shape and 3-D motion, of a rigid object from a two-dimensional (2-D) video sequence. The main ingredients of our approach are as follows: 1) we describe the unknown shape of the 3-D rigid object by polynomial patches; 2) projections of these patches in the image plane move according to parametric 2-D motion models; 3) we recover the parameters describing the 3-D shape and 3-D motion from the 2-D motion parameters by factorizing a matrix that is rank 1 in a noiseless situation. Our method is simultaneously an extension and a simplification of the original factorization method of Tomasi and Kanade (1992). We track regions where the 2-D motion in the image plane is described by a single set of parameters, avoiding the need to track a large number of pointwise features, in general, a difficult task. Then our method estimates the parameters describing the 3-D structure by factoring a rank 1 matrix, not rank 3 as in Tomasi and Kanade. This allows the use of fast iterative algorithms to compute the 3-D structure that best fits the data. Experimental results with real-life video sequences illustrate the good performance of our approach