A flexible mixed-signal image processing pipeline using 3D chip stacks

This work presents a highly flexible mixed-signal CMOS image sensor suitable for smart camera applications. These systems need to fit different constraints regarding power consumption, speed and quality, and the optimal compromise may differ depending on the application. Moreover, the best implementation of a desired image processing task may be in the analog or the digital domain, or even a combined computation. Different aspects starting from the image sensor and signal acquisition up to the pre-processing in analog and digital domain are investigated in this paper to optimize not just one part of the system, but the whole system altogether. Moreover, it is shown that analog processing algorithms can improve signal quality, processing speed and latency while being able to save power, which is important for real-time systems. In order to be able to carry out spatial operations, the state-of-the-art sensor is modified to be able to read out multiple pixels at the same time. This allows analog spatial filter operations which consume significantly less power. As an example, an averaging filter is described which needs less than 5.3 % of the power–time product of a digital implementation for one computation. To enhance data throughput and flexibility, 3D chip stacking is proposed to partition the sensor in smaller units and enable massively parallel processing.     

3D geometry reconstruction from orthographic views: A method based on 3D image processing and data fitting

Industrial esthetic designers typically produce hand-drawn sketches in the form of orthographic projections. A subsequent translation from 2D-drawings to 3D-models is usually necessary. This involves a considerably time consuming process, so that some automation is advisable.     

3D finger tracking and recognition image processing for real-time music playing with depth sensors

In this research, we propose a state-of-the-art 3D finger gesture tracking and recognition method. We use the depth sensors for both hands in real time music playing. In line with the development of 3D depth cameras, we implemented a set of 3D gesture-based instruments, such as Virtual Cello and Virtual Piano, which need precise finger tracking in 3D space. For hands tracking, model-based tracking for left hand and appearance-based tracking for right hand techniques are proposed. To detect finger gestures, our approaches consist number of systematic steps as reducing noise in depth map and geometrical processing for Virtual Cello. For Virtual Piano, we introduce the Neural Network (NN) method to detect special hand gestures. It has Multilayer Perceptron (MLP) structure with back propagation training. Literature has few examples using touch screen as medium, with fixed-coordinates, and 2D–gestures to control MIDI input. The end users should no longer carry anything on their hands. We use Senz3D and Leap Motion due to a few technical benefits. Senz3D and Leap Motion use a closer distance to hands, thus detailed finger gestures can be precisely identified. In the past years, we announced a set of virtual musical instruments and the MINE Virtual Band. Our research work is tested on lab environment and professional theatrical stage. More information and demonstrations of the proposed method can be accessed at: http://video.minelab.tw/DETS/VMIB/.     

A method for expressing human posture as 3DCG using thermal image processing and 3D model fitting

An imitation of human motion has been used as a promising technique for the development of a robot. Some techniques such as motion capture systems and data-gloves are used for analyzing human motion. However, since those methods involve (a) environmental restrictions such as the preparation of two or more cameras and the strict control of brightness, and (b) physical restrictions such as the wearing of markers and/or data-gloves, they are far removed from a method for recognizing human motion in a natural condition. In this article, we propose a method that makes a 3-dimensional CG (3DCG) by transforming a feature vector of human posture on a thermal image into a 3DCG model. The 3DCG models for use as training data are made with manual model fitting. Then human models synthesized by our method are geometrically evaluated in CG space. The average error in position is about 10 cm. Such a relatively small error might be acceptable in some cases e.g., 3DCG animation generation and the imitation of human motion by a robot. Our method has neither physical nor environmental restrictions. The rotation-angles at each joint obtained by our method can be used for an imitation of human posture by a robot.     

基于单张图像的三维人脸重建

利用一种基于法线的模型变形方法,从单张图像重建高质量的三维人脸.利用球谐函数和一个初始参考模型计算得到模型上每个顶点的法线,利用法线使参考模型变形.实验结果表明:提出的算法可以从单幅图像重建具有细节的高质量三维人脸.     

2D/3D image registration using regression learning

In computer vision and image analysis, image registration between 2D projections and a 3D image that achieves high accuracy and near real-time computation is challenging. In this paper, we propose a novel method that can rapidly detect an object’s 3D rigid motion or deformation from a 2D projection image or a small set thereof. The method is called CLARET (Correction via Limited-Angle Residues in External Beam Therapy) and consists of two stages: registration preceded by shape space and regression learning. In the registration stage, linear operators are used to iteratively estimate the motion/deformation parameters based on the current intensity residue between the target projection(s) and the digitally reconstructed radiograph(s) (DRRs) of the estimated 3D image. The method determines the linear operators via a two-step learning process. First, it builds a low-order parametric model of the image region’s motion/deformation shape space from its prior 3D images. Second, using learning-time samples produced from the 3D images, it formulates the relationships between the model parameters and the co-varying 2D projection intensity residues by multi-scale linear regressions. The calculated multi-scale regression matrices yield the coarse-to-fine linear operators used in estimating the model parameters from the 2D projection intensity residues in the registration. The method’s application to Image-guided Radiation Therapy (IGRT) requires only a few seconds and yields good results in localizing a tumor under rigid motion in the head and neck and under respiratory deformation in the lung, using one treatment-time imaging 2D projection or a small set thereof.     

2D/3D image registration on the GPU

We present a method that performs a rigid 2D/3D image registration efficiently on the Graphical Processing Unit (GPU). As one main contribution of this paper, we propose an efficient method for generating realistic DRRs that are visually similar to x-ray images. Therefore, we model some of the electronic post-processes of current x-ray C-arm-systems. As another main contribution, the GPU is used to compute eight intensity-based similarity measures between the DRR and the x-ray image in parallel. A combination of these eight similarity measures is used as a new similarity measure for the optimization. We evaluated the performance and the precision of our 2D/3D image registration algorithm using two phantom models. Compared to a CPU + GPU algorithm, which calculates the similarity measures on the CPU, our GPU algorithm is between three and six times faster. In contrast to single similarity measures, our new similarity measure achieved precise and robust registration results for both phantom models.     

Analysis of all-in-focus image processing in image restoration

We discuss a new deblurring problems in this paper. Focus measurements play a fundamental role in image processing techniques. Most traditional methods neglect spatial information in the frequency domain. Therefore, this study analyzed image data in the frequency domain to determine the value of spatial information. but instead misleading noise reduction results . We found that the local feature is not always a guide for noise reduction. This finding leads to a new method to measure the image edges in focus deblurring. We employed an all-in-focus measure in the frequency domain, based on the energy level of frequency components. We also used a multi-circle enhancement model to analyze this spatial information to provide a more accurate method for measuring images. We compared our results with those using other methods in similar studies. Findings demonstrate the effectiveness of our new method.     

A theoretical 2D image model for locating 3D targets

To construct a water quality monitoring system, challenging issues need to be addressed regarding the acquisition of target information (e.g. 3D location and occlusion) as well as the behavioural analysis of aquatic organisms. This paper presents a novel 3D information acquisition and location method, by means of an information acquisition platform consisting of a monitoring terminal, frame grabbers, a single camera and a single mirror. Using this platform, we propose a theoretical 2D image model for locating 3D targets and then validate it using data obtained from both real and artificial fish. The proposed model is based on the principles of light refraction, plane mirror imaging, underwater objects and camera imaging as well as the technologies of digital to analog conversion and object segmentation. In contrast with existing methods, our method can accurately reflect 3D information of aquatic organisms, thus providing critical technical support for the development of water quality monitoring systems in the future.     

面向图像三维重建的无人机航线规划

随着无人机技术的发展,无人机序列影像三维重建越来越受到人们的关注。为完整重建任务区域的三维模型并减少无人机飞行功耗,提出一种面向图像三维重建的无人机航线规划算法。针对凸多边形任务区域,在图像重叠度和时间连续性的要求下,基于光栅法规划扫描航线并结合最佳扫描方向使得转弯次数最少。借助Gazebo仿真平台,对比验证了无人机按照该算法规划的航线飞行时功耗更小,且拍摄得到的序列影像能够重建任务区域三维模型。     

Estimating 3D egomotion from perspective image sequence

The computation of sensor motion from sets of displacement vectors obtained from consecutive pairs of images is discussed. The problem is investigated with emphasis on its application to autonomous robots and land vehicles. The effects of 3D camera rotation and translation upon the observed image are discussed, particularly the concept of the focus of expansion (FOE). It is shown that locating the FOE precisely is difficult when displacement vectors are corrupted by noise and errors. A more robust performance can be achieved by computing a 2D region of possible FOE locations (termed the fuzzy FOE) instead of looking for a single-point FOE. The shape of this FOE region is an explicit indicator of the accuracy of the result. It has been shown elsewhere that given the fuzzy FOE, a number of powerful inferences about the 3D sense structure and motion become possible. Aspects of computing the fuzzy FOE are emphasized, and the performance of a particular algorithm on real motion sequences taken from a moving autonomous land vehicle is shown     

RGB-D image saliency detection from 3D perspective

Multimedia Tools and Applications - With the advent of stereo camera saliency object detection for RGB-D image is attracting more and more interest. Most existing algorithms treat RGB-D image as...     

资源三号卫星CCD影像云处理方法研究

在分析目前较为常用的多种去云方法特点后,选取改进的同态滤波算法进行去云处理,并与同态滤波法进行对比研究,通过定性分析与定量评价两方面比较。结果表明:基于改进同态滤波法对资源三号融合影像去云处理产生的遥感图像失真较小,同时很大程度地保持了空间纹理细节信息,是一种适合于资源三号影像去云的较好方法。