Geometric techniques for 3D tracking of ultrasound sensor,tumor segmentation in ultrasound images,and 3D reconstruction

This paper presents different methods, some based on geometric algebra, for ultrasound probe tracking in endoscopic images, 3D allocation of the ultrasound probe, ultrasound image segmentation (to extract objects like tumors), and 3D reconstruction of the surface defined by a set of points. The tracking of the ultrasound probe in endoscopic images is done with a particle filter and an auxiliary method based on thresholding in the HSV space. The 3D pose of the ultrasound probe is calculated using conformal geometric algebra (to locate each slide in 3D space). Each slide (ultrasound image) is segmented using two methods: the level-set method and the morphological operators approach in order to obtain the object we are interested in. The points on the object of interest are obtained from the segmented ultrasound images, and then a 3D object is obtained by refining the convex hull. To do that, a peeling process with an adaptive radius is applied, all of this in the geometric algebra framework. Results for points from ultrasound images, as well as for points from objects from the AimatShape Project, are presented (A.I.M.A.T.S.H.A.P.E. – Advanced an Innovative Models And Tools for the development of Semantic-based systems for Handling, Acquiring, and Processing knowledge Embedded in multidimensional digital objects).     

乒乓球的三维运动轨迹重建

主要基于图像序列对乒乓球的运动轨迹进行三维重建,并对乒乓球运动形态进行分析.首先对采集的图像进行立体校正,利用颜色识别和改进的霍夫圆检测算法提取出序列图像中乒乓球的圆心坐标;然后根据前后帧图像的特征点坐标差值在时间序列上匹配特征点;最后,利用三角测量法对匹配的特征点进行三维重建,并计算出乒乓球不同时刻的速度和加速度,实现了动态物体的三维运动重建.实验结果表明该三维运动重建方法提高了特征提取的准确性,有效地实现了时间序列上的匹配,获得了物体的三维运动数据.     

利用序列ISAR图像获取空间目标3-D信息的方法

如何从空间目标序列性二维(2-D,Two-Dimentional)逆合成孔径雷达(ISAR,Inverse Synthetic Aperture Radar)成像获取目标的三维(3-D)信息,是目标特征自动识别(ATR,Automatic Target Recognition)技术的重要研究课题。利用双向射线跟踪(BART,Bidirectional Analytic Ray Tracing)方法,计算连续多角度观测条件下空间目标的电磁散射数据,并由此获取空间目标的ISAR序列2-D图像。再利用KLT(Kanade-Lucas-Tomasi)特征跟踪算法,跟踪提取2\|D序列ISAR图像中的特征点(强散射点),获得其2-D坐标。然后,基于正交因式分解法(OFM,Orthographic Factorization Method),计算强散射点的3\|D坐标,获取空间目标的3-D信息。通过简单六棱柱模型,验证重构算法的精度;并以ENVISAT卫星模型为例,给出强散射点的3-D重构结果。结果表明,本文对空间目标3\|D信息获取方法能有效地从ISAR序列2-D图像中重构目标的三维信息。     

Predicting Object Dynamics From Visual Images Through Active Sensing Experiences

Prediction of dynamic features is an important task for determining the manipulation strategies of an object. This paper presents a technique for predicting dynamics of objects relative to the robot's motion from visual images. During the training phase, the authors use the recurrent neural network with parametric bias (RNNPB) to self-organize the dynamics of objects manipulated by the robot into the PB space. The acquired PB values, static images of objects and robot motor values are input into a hierarchical neural network to link the images to dynamic features (PB values). The neural network extracts prominent features that each induce object dynamics. For prediction of the motion sequence of an unknown object, the static image of the object and robot motor value are input into the neural network to calculate the PB values. By inputting the PB values into the closed loop RNNPB, the predicted movements of the object relative to the robot motion are calculated recursively. Experiments were conducted with the humanoid robot Robovie-IIs pushing objects at different heights. The results of the experiment predicting the dynamics of target objects proved that the technique is efficient for predicting the dynamics of the objects.     

利用信息融合方法的三维特征提取

本文提出了一种基于信息融合的物体三维特征的提取方法，该方法利用两幅互相配准的三维测距图像和灰度图像，来提取多面体的三维特征。首先，通过分析灰度图像中的灰度变化及测距图像中的测距值变化，分别求取各自图像中物体的特征点及特征边；然后，利用两配准图像之间的对应关系，求得所有特征点、面与多边形在三维测距图像中的三维表示；接着，通过分析三维测距图像中所测得的各候选平面上特定点与边处的曲率及法向，验证候选平面     

The Alignment of Objects with Smooth Surfaces

This paper examines the recognition of rigid objects bounded by smooth surfaces, using an alignment approach. The projected image of such an object changes during rotation in a manner that is generally difficult to predict. An approach to this problem is suggested, using the 3D surface curvature at the points along the silhouette. The curvature information requires a single number for each point along the object′s silhouette, the radial curvature at the point. We have implemented this method and tested it on images of complex 3D objects. Models of the viewed objects were acquired using three images of each object. The implemented scheme was found to give accurate predictions of the objects′ appearances for large transformations. Using this method, a small number of (viewer-centered) models can be used to predict the new appearance of an object from any given viewpoint.     

Acquiring linear subspaces for face recognition under variable lighting

Previous work has demonstrated that the image variation of many objects (human faces in particular) under variable lighting can be effectively modeled by low-dimensional linear spaces, even when there are multiple light sources and shadowing. Basis images spanning this space are usually obtained in one of three ways: a large set of images of the object under different lighting conditions is acquired, and principal component analysis (PCA) is used to estimate a subspace. Alternatively, synthetic images are rendered from a 3D model (perhaps reconstructed from images) under point sources and, again, PCA is used to estimate a subspace. Finally, images rendered from a 3D model under diffuse lighting based on spherical harmonics are directly used as basis images. In this paper, we show how to arrange physical lighting so that the acquired images of each object can be directly used as the basis vectors of a low-dimensional linear space and that this subspace is close to those acquired by the other methods. More specifically, there exist configurations of k point light source directions, with k typically ranging from 5 to 9, such that, by taking k images of an object under these single sources, the resulting subspace is an effective representation for recognition under a wide range of lighting conditions. Since the subspace is generated directly from real images, potentially complex and/or brittle intermediate steps such as 3D reconstruction can be completely avoided; nor is it necessary to acquire large numbers of training images or to physically construct complex diffuse (harmonic) light fields. We validate the use of subspaces constructed in this fashion within the context of face recognition.     

基于对称物体的3维重构技术研究

3维重构理论与技术是计算机视觉领域最重要的热点问题之一,而基于单幅图像的3维重构由于缺乏足够的几何信息而难以达到预期效果,已成为世纪性难题。针对大部分物体具有对称性特征,或可分解为有限个对称物体元的客观事实,提出基于透视逆变换原理,首先建立包含摇角、倾角、摆角等三元的透视变换矩阵T;再由链码表示的物体轮廓提取特征直线,根据平行线束投影角相近性特点求解主灭点信息,进而确定视点位置、物体对称平面;根据对称性特征,利用物体假想对称平面,通过人工交互指定3对已知对称点的图像坐标及其对应点的空间坐标确定透视变换矩阵T,继而反求物体表面其他特征点的空间位置,最后利用OpenGL软件包实现物体3维模型的重建。     

3D Image Synthesis for B-Reps Objects

This paper presents a new algorithm for generating 3D images of B-reps objects with trimmed surface boundaries.The 3D image is a discrete voxel-map representation within a Cubic Frame Buffer (CFB).The definition of 3D images for curve,surface and solid object are introduced which imply the connectivity and fidelity requirements.Adaptive Forward Differencing matrix (AFD-matrix) for 1D-3D manifolds in 3D space is developed.By setting rules to update the AFD-matrix,the forward difference direction and stepwise can be adjusted.Finally,an efficient algorithm is presented based on the AFD-matrix concept for converting the object in 3D space to 3D image in 3D discrete space.     

In the wild image retrieval and clustering for 3D cultural heritage landmarks reconstruction

One of the main characteristics of Internet era is the free and online availability of extremely large collections of images located on distributed and heterogeneous platforms over the web. The proliferation of millions of shared photographs spurred the emergence of new image retrieval techniques based not only on images’ visual information, but on geo-location tags and camera exif data. These huge visual collections provide a unique opportunity for cultural heritage documentation and 3D reconstruction. The main difficulty, however, is that the internet image datasets are unstructured containing many outliers. For this reason, in this paper a new content-based image filtering is proposed to discard image outliers that either confuse or significantly delay the followed e-documentation tools, such as 3D reconstruction of a cultural heritage object. The presented approach exploits and fuses two unsupervised clustering techniques: DBSCAN and spectral clustering. DBSCAN algorithm is used to remove outliers from the initially retrieved dataset and spectral clustering discriminate the noise free image dataset into different categories each representing characteristic geometric views of cultural heritage objects. To discard the image outliers, we consider images as points onto a multi-dimensional manifold and the multi-dimensional scaling algorithm is adopted to relate the space of the image distances with the space of Gram matrices through which we are able to compute the image coordinates. Finally, structure from motion is utilized for 3D reconstruction of cultural heritage landmarks. Evaluation on a dataset of about 31,000 cultural heritage images being retrieved from internet collections with many outliers indicate the robustness and cost effectiveness of the proposed method towards a reliable and just-in-time 3D reconstruction than existing state-of-the-art techniques.     

Video image target monitoring based on RNN-LSTM

Traditional image object classification and detection algorithms and strategies cannot meet the problem of video image acquisition and processing. Deep learning deliberately simulates the hierarchical structure of human brain, and establishes the mapping from low-level signals to high-level semantics, so as to achieve hierarchical feature representation of data. Deep learning technology has powerful visual information processing ability, which has become the forefront technology and domestic and international research hotspots to deal with this challenge. In order to solve the problem of target space location in video surveillance system, time-consuming and other problems, in this paper, we propose the algorithm based on RNN-LSTM deep learning. At the same time, according to the principle of OpenGL perspective imaging and photogrammetry consistency, we use 3D scene simulation imaging technology, relying on the corresponding relationship between video images and simulation images we locate the target object. In the 3D virtual scene, we set up the virtual camera to simulate the imaging processing of the actual camera, and the pixel coordinates in the video image of the surveillance target are substituted into the simulation image, next, the spatial coordinates of the target are inverted by the inverse process of the virtual imaging. The experimental results show that the detection of target objects has high accuracy, which has an important reference value for outdoor target localization through video surveillance images.

     

Parallel distributed detection of feature trajectories in multiplediscontinuous motion image sequences

Concerns the 3D interpretation of image sequences showing multiple objects in motion. Each object exhibits smooth motion except at certain time instants when a motion discontinuity may occur. The objects are assumed to contain point features which are detected as the images are acquired. Estimating feature trajectories in the first two frames amounts to feature matching. As more images are acquired, existing trajectories are extended. Both initial detection and extension of trajectories are done by enforcing pertinent constraints from among the following: similarity of the image plane arrangement of neighboring features, smoothness of the 3D motion and smoothness of the image plane motion. The constraints are incorporated into energy functions which are minimized using 2D Hopfield networks. Wrong matches that result from convergence to local minima are eliminated using a 1D Hopfield-like network. Experimental results on several image sequences are shown.     

A New Approach to Pose Detection using a Trinocular Stereovision System

Object location and tracking is a major issue in computer vision. This problem is normally solved through the extraction of representative features of the object, and the two-dimensional coordinates of these image features are used to compute the position of the object. When more than one camera is used, a certain similarity measure between the image features extracted from both stereoscopic images helps to match the correspondences. In this way, three-dimensional measurements can be recovered from the 2D coordinates of the features extracted from different cameras. In this paper the use of a trinocular system is considered to estimate both the position and velocity of known objects by using their apparent area, and with no use of the image-plane coordinates of the object 's features. A high precision low-level image processor has been developed for performing object labeling and noise filtering of the images at video rate. Then, a position measurement tool uses the apparent area captured by every camera to locate the object. This enables us to estimate the position of the object. Finally, a prediction tool refines the estimation in locating the object. We show the performance of the trinocular system with a real implementation. This system has been designed to process the images provided by any conventional of high-speed cameras at video rate.     

An object expression system using depth-maps

The current of Augmented Reality are data gloves or markers used for interactions between object and background. But, this results in the inconvenience in use and reduced immersiveness. To reinforce immersiveness in AR, added input devices should be removed. To this end, spatial coordinates should be accurately perceived even when a marker has been attached. In this paper, an object expression system was proposed that uses depth-maps for interactions without any additional input devices in order to improve immersiveness. Immersiveness was improved by projecting obtained images on 2D spaces, extracting vanishing lines, calculating the virtual spatial coordinates of the projected images, and varying the sizes of the inserted objects in accordance with the sizes of the areas of virtual coordinates, based on the images projected on the 2D coordinates. By using this system, the use of 3D modelers could be excluded when 3D objects were created; thus, the efficiency of object creation could be improved.     

激光点云和数字影像结合的目标重建研究

针对复杂目标的重建,利用近景激光扫描仪和机控的旋转平台获取目标的激光点云与数字影像两种不同的数据源,提出了一种大旋转角度的不同视点激光点云的配准算法,并实现了多视点云的无缝配准,建立目标完整的三维几何模型,对实验结果进行了精度评定。在此基础上,利用序列数字影像实现目标的纹理恢复,使得重建的目标既能够表现出精确的几何特征,又能够表现出丰富的纹理特征。实验证明,复杂目标重建的几何精度和纹理效果与实际量测结果和人的主观视觉感知具有良好的一致性。     

Spatially varying image based lighting by light probe sequences

We present a novel technique for capturing spatially or temporally resolved light probe sequences, and using them for image based lighting. For this purpose we have designed and built a real-time light probe, a catadioptric imaging system that can capture the full dynamic range of the lighting incident at each point in space at video frame rates, while being moved through a scene. The real-time light probe uses a digital imaging system which we have programmed to capture high quality, photometrically accurate color images of 512×512 pixels with a dynamic range of 10000000:1 at 25 frames per second. By tracking the position and orientation of the light probe, it is possible to transform each light probe into a common frame of reference in world coordinates, and map each point and direction in space along the path of motion to a particular frame and pixel in the light probe sequence. We demonstrate our technique by rendering synthetic objects illuminated by complex real world lighting, first by using traditional image based lighting methods and temporally varying light probe illumination, and second an extension to handle spatially varying lighting conditions across large objects and object motion along an extended path.     

基于水动力学、分数阶微分及Steger算法的线状目标提取

目的 线状目标的检测具有非常广泛的应用领域,如车道线、道路及裂缝的检测等,而裂缝是其中最难检测的线状目标。为避免直接提取线状目标时图像分割难的问题,以裂缝和车道线为例,提出了一种新的跟踪线状目标中线的算法。方法 对图像进行高斯平滑,用一种新的分数阶微分模板增强图像中的模糊及微细线状目标;基于Steger算法提出一种提取线状目标中心线特征点的算法,避免了提取整体目标的困难;根据水动力学思想将裂隙看成溪流,通过最大熵阈值处理后,先进行特征点的连接,再基于线段之间的距离及夹角进行线段之间的连接（溪流之间的融合）。结果 对300幅裂缝图像及4种类别的其他线状目标图像进行试验,并与距离变换、最大熵阈值法+细线化Otsu阈值分割+细线化、谷底边界检测等类似算法进行比较分析,本文算法检测出的线状目标的连续性好、漏检（大间隙少）和误检（毛刺及多余线段少）率均较低。结论 本文算法能够在复杂的线状目标图像中准确快速地提取目标的中心线,一定程度上改善了复杂线状目标图像分割难的问题。     

仿人形机器人火炬传递中的高精度目标特征提取

以仿人形机器人的火炬传递为背景,针对立体视觉系统的图像处理展开研究,提出了一种基于色标的高精度特征提取方法.利用火炬上的矩形色标,在RGB空间基于色彩进行图像分割,并利用K-L变换对边缘点进行分组,通过Hough变换和最小二乘法对色标边缘进行直线拟合,得到矩形色标顶点的高精度图像坐标,为提高火炬位姿测量的精度提供了基础.实验结果验证了本文方法的有效性.     

Automatic 3D object segmentation in multiple views using volumetric graph-cuts

We propose an algorithm for automatically obtaining a segmentation of a rigid object in a sequence of images that are calibrated for camera pose and intrinsic parameters. Until recently, the best segmentation results have been obtained by interactive methods that require manual labelling of image regions. Our method requires no user input but instead relies on the camera fixating on the object of interest during the sequence. We begin by learning a model of the object’s colour, from the image pixels around the fixation points. We then extract image edges and combine these with the object colour information in a volumetric binary MRF model. The globally optimal segmentation of 3D space is obtained by a graph-cut optimisation. From this segmentation an improved colour model is extracted and the whole process is iterated until convergence.