Three-dimensional image sensing and reconstruction with time-division multiplexed computational integral imaging

A method to compute high-resolution three-dimensional images based on integral imaging is presented. A sequence of integral images (IIs) is captured by means of time-division multiplexing with a moving lenslet array technique. For the acquisition of each II, the location of the lenslet array is shifted periodically within the lenslet pitch in a plane perpendicular to the optical axis. The II sequence obtained by the detector array is processed digitally with superresolution reconstruction algorithms to obtain a reconstructed image, appropriate to a viewing direction, which has a spatial resolution beyond the optical limitation.     

Signal-processing approaches for image-resolution restoration for TOMBO imagery

Thin observation module by bounded optics (TOMBO) is an optical system that achieves compactness and thinness by replacing a conventional large full aperture by a lenslet array with several smaller apertures. This array allows us to collect diverse low-resolution measurements. Finding an efficient way of combining these diverse measurements to make a high-resolution image is an important research problem. We focus on finding a computational method for performing the resolution restoration and evaluating the method via simulations. Our approach is based on advanced signal-processing concepts: we construct a computational data model based on Fourier optics and propose restoration algorithms based on minimization of an information-theoretic measure, called Csiszár's I divergence between two nonnegative quantities: the measured data and the hypothetical images that are induced by our algorithms through the use of our computational data model. We also incorporate Poisson and Gaussian noise processes to model the physical measurements. To solve the optimization problem, we adapt the popular expectation-maximization method. These iterative algorithms, in a multiplicative form, preserve powerful nonnegativity constraints. We further incorporate a regularization based on minimization of total variation to suppress incurring artifacts such as roughness on the surfaces of the estimates. Two sets of simulation examples show that the algorithms can produce very high-quality estimates from noiseless measurements and reasonably good estimates from noisy measurements, even when the measurements are incomplete. Several interesting and useful avenues for future work such as the effects of measurement selection are suggested in our conclusional remarks.     

Contour superresolved imaging of static ground targets using satellite platform

We propose a method for increasing the contour resolution of static ground targets and to overcome the diffraction limit of an optical system installed on top of a satellite. The resolution improvement is obtained by using a sequence of low-resolution images taken from different angles realized by the movement of the satellite platform. The superresolving process is obtained by the generation of relative movement between the inspected object and the a priori known high-resolution background. The relative movement is caused because the images are taken from different angles. The captured set of low-resolution images are decoded by the a priori known high-resolution background obtained from a set of reference images taken only once by a high-resolution camera. The proposed concept is demonstrated via Matlab simulation and laboratory experiments.     

High-Resolution Digital Integral Photography by use of a Scanning Microlens Array

We suggest what we believe is a new three-dimensional (3-D) camera system for integral photography. Our method enables high-resolution 3-D imaging. In contrast to conventional integral photography, a moving microlens array (MLA) and a low-resolution camera are used. The intensity distribution in the MLA image plane is sampled sequentially by use of a pinhole array. The inversion problem from pseudoscopic to orthoscopic images is dealt with by electronic means. The new method is suitable for real-time 3-D imaging. We verified the new method experimentally. Integral photographs with a resolution of 3760 pixels x 2560 pixels (188 x 128 element images) are presented.     

Integral imaging using phase-only LCoS spatial light modulators as Fresnel lenslet arrays

We present a digital integral imaging system. A Fresnel lenslet array pattern is written on a phase-only LCoS spatial light modulator device (SLM) to replace the regular analog lenslet array in a conventional integral imaging system. We theoretically analyze the capture part of the proposed system based on Fresnel wave propagation formulation. Because of pixelation and quantization of the lenslet array pattern, higher diffraction orders and multiple focal points emerge. Because of the multiple focal planes introduced by the discrete lenslets, multiple image planes are observed. The use of discrete lenslet arrays also causes some other artifacts on the recorded elemental images. The results reduce to those available in the literature when the effects introduced by the discrete nature of the lenslets are omitted. We performed simulations of the capture part. It is possible to obtain the elemental images with an acceptable visual quality. We also constructed an optical integral imaging system with both capture and display parts using the proposed discrete Fresnel lenslet array written on a SLM. Optical results when self-luminous objects, such as an LED array, are used indicate that the proposed system yields satisfactory results.     

Multiaperture imaging

We study the reconstruction of a high-resolution image from multiple low-resolution images by using a nonlinear iterative backprojection algorithm. We exploit diversities in the imaging channels, namely, the number of imagers, magnification, position, rotation, and fill factor, to undo the degradation caused by the optical blur, pixel blur, and additive noise. We quantify the improvements gained by these diversities in the reconstruction process and discuss the trade-off among system parameters. As an example, for a system in which the pixel size is matched to the diffraction-limited optical blur size at a moderate detector noise level, we can reduce the reconstruction root-mean-square error by 570% by using 16 cameras and a large amount of diversity. The algorithm was implemented on a 56 camera array specifically constructed to demonstrate the resolution-enhancement capabilities. Practical issues associated with building and operating this device are presented and analyzed.     

High speed preprocessing system

In systems employing tracking, the area of interest is recognized using a high resolution camera and is handed over to the low resolution receiver. The images seen by the low resolution receiver and by the operator through the high resolution camera are different in spatial resolution. In order to establish the correlation between these two images, the high-resolution camera image needs to be preprocessed and made similar to the low-resolution receiver image. This paper discusses the implementation of a suitable preprocessing technique, emphasis being given to develop a system both in hardware and software to reduce processing time. By applying different software/hardware techniques, the execution time has been brought down from a few seconds to a few milli-seconds for a typical set of conditions. The hardware is designed around i486 processors and software is developed in PL/M. The system is tested to match the images obtained by two different sensors of the same scene. The hardware and software have been evaluated with different sets of images.     

Analysis and application of multiframe superresolution processing for conventional imaging systems and lenslet arrays

Low-cost compact sensors for ultrasmall systems are a pressing need in many new applications. One potential solution is a shallow aspect ratio system using a lenslet array to form multiple undersampled subimages of a scene on a single focal plane array, where processing techniques then produce an upsampled restored image. We have investigated the optimization and theoretical limits of the performance of such arrays. We have built a hardware simulator and developed algorithms to process imagery similar to that of a full lenslet imaging sensor, which allowed us to quickly test optical components, algorithms, and complete system designs for future lenslet imaging systems.     

Superresolution image reconstruction from a sequence of aliased imagery

We present a superresolution image reconstruction from a sequence of aliased imagery. The subpixel shifts (displacement) among the images are unknown due to the uncontrolled natural jitter of the imager. A correlation method is utilized to estimate subpixel shifts between each low-resolution aliased image with respect to a reference image. An error-energy reduction algorithm is derived to reconstruct the high-resolution alias-free output image. The main feature of this proposed error-energy reduction algorithm is that we treat the spatial samples from low-resolution images that possess unknown and irregular (uncontrolled) subpixel shifts as a set of constraints to populate an oversampled (sampled above the desired output bandwidth) processing array. The estimated subpixel locations of these samples and their values constitute a spatial domain constraint. Furthermore, the bandwidth of the alias-free image (or the sensor imposed bandwidth) is the criterion used as a spatial frequency domain constraint on the oversampled processing array. The results of testing the proposed algorithm on the simulated low- resolution forward-looking infrared (FLIR) images, real-world FLIR images, and visible images are provided. A comparison of the proposed algorithm with a standard interpolation algorithm for processing the simulated low-resolution FLIR images is also provided.     

Adaptive framework for robust high-resolution image reconstruction in multiplexed computational imaging architectures

In multiplexed computational imaging schemes, high-resolution images are reconstructed by fusing the information in multiple low-resolution images detected by a two-dimensional array of low-resolution image sensors. The reconstruction procedure assumes a mathematical model for the imaging process that could have generated the low-resolution observations from an unknown high-resolution image. In practical settings, the parameters of the mathematical imaging model are known only approximately and are typically estimated before the reconstruction procedure takes place. Violations to the assumed model, such as inaccurate knowledge of the field of view of the imagers, erroneous estimation of the model parameters, and/or accidental scene or environmental changes can be detrimental to the reconstruction quality, even if they are small in number. We present an adaptive algorithm for robust reconstruction of high-resolution images in multiplexed computational imaging architectures. Using robust M-estimators and incorporating a similarity measure, the proposed scheme adopts an adaptive estimation strategy that effectively deals with violations to the assumed imaging model. Comparisons with nonadaptive reconstruction techniques demonstrate the superior performance of the proposed algorithm in terms of reconstruction quality and robustness.     

Generic camera model and its calibration for computational integral imaging and 3D reconstruction

Integral imaging (II) is an important 3D imaging technology. To reconstruct 3D information of the viewed objects, modeling and calibrating the optical pickup process of II are necessary. This work focuses on the modeling and calibration of an II system consisting of a lenslet array, an imaging lens, and a charge-coupled device camera. Most existing work on such systems assumes a pinhole array model (PAM). In this work, we explore a generic camera model that accommodates more generality. This model is an empirical model based on measurements, and we constructed a setup for its calibration. Experimental results show a significant difference between the generic camera model and the PAM. Images of planar patterns and 3D objects were computationally reconstructed with the generic camera model. Compared with the images reconstructed using the PAM, the images present higher fidelity and preserve more high spatial frequency components. To the best of our knowledge, this is the first attempt in applying a generic camera model to an II system.     

Algorithm to increase the largest aberration that can be reconstructed from Hartmann sensor measurements

Conventional Hartmann sensor processing relies on locating the centroid of the image that is formed behind each element of a lenslet array. These centroid locations are used for computing the local gradient of the incident aberration, from which the phase of the incident wave front is calculated. The largest aberration that can reliably be sensed in a conventional Hartmann sensor must have a local gradient small enough that the spot formed by each lenslet is confined to the area behind the lenslet: If the local gradient is larger, spots form under nearby lenslets, causing a form of cross talk between the wave-front sensor channels. We describe a wave-front reconstruction algorithm that processes the whole image measured by a Hartmann sensor and a conventional image that is formed by use of the incident aberration. We show that this algorithm can accurately estimate aberrations for cases in which the aberration is strong enough to cause many of the images formed by individual lenslets to fall outside the local region of the Hartmann sensor detector plane defined by the edges of a lenslet.     

Real-time pickup method for a three-dimensional image based on integral photography

We studied integral photography (IP), which creates three-dimensional autostereoscopic images. In particular we studied the possibility of a new method that uses a television camera to shoot directly numerous real images produced by a lens array. Unlike the conventional IP method in which the film is placed immediately behind a lens array, this method employs a television camera, which enables us to shoot moving pictures. Of a number of factors affecting the process of image pickup, we examined some optical factors and compared them with those obtained by the conventional IP method. The results show that with this new direct pickup method that uses a television camera, we can obtain an IP image like those obtained by using the conventional IP method. Further, we conducted an experiment with an high-definition TV camera, confirming the production of an autostereoscopic image by using a display device that combines a liquid-crystal panel and pinholes.     

利用二维插值核的高分辨力图像重构

利用连续图像采样及其内插公式,提出了一种基于二维插值核重建高分辨力图像的新算法。该算法先使一般低分辨力序列图像具有标准位移关系,再根据高、低分辨力图像之间的关系重构高分辨力图像。同时利用正则化技术导出了一套重复迭代算法,以提高算法的处理效果和克服病态问题。实验表明,该算法使图像的细节和清晰度大大增强,分辨力大幅度提高,明显优于零阶保持插值算法的处理结果,其峰值信噪比提高8-9dB。     

Thermographic Imaging for High-Temperature Composite Materials—A Defect Detection Study

The ability of a thermographic imaging technique for detecting flat-bottom hole defects of various diameters and depths was evaluated in four composite systems (two types of ceramic-matrix composites, one metal-matrix composite, and one polymer-matrix composite) of interest as high-temperature structural materials. The holes ranged from 1 to 13 mm in diameter and 0.1 to 2.5 mm in depth in samples approximately 2—3 mm thick. The thermographic imaging system utilized a scanning mirror optical system and infrared (IR) focusing lens in conjunction with a mercury—cadmium—telluride infrared detector element to obtain high-resolution infrared images. High-intensity flash lamps located on the same side as the infrared camera were used to heat the samples. After heating, up to 30 images were sequentially acquired at 70—150 ms intervals. Limits of detectability based on depth and diameter of the flat-botton holes were defined for each composite material. Ultrasonic and radiographic images of the samples were obtained and compared with the thermographic images. This study was done under a nonreimbursable Space Act Agreement between NASA—Lewis Research Center and Bales Scientific, Inc., to allow several heating configurations to be evaluated in a cost-effective and timely fashion.     

Multichanneled finite-conjugate imaging

Multichanneled imaging systems rely on nonredundant images recorded by an array of low-resolution imagers to enable construction of a high-resolution image. We show how the varying degree of redundancy associated with imaging throughout the imaged volume effects image quality. Using ray-traced image simulations and a metric used as a proxy for human perception, we show that robust recovery of high-resolution images can be obtained by avoiding excessive redundancy and that this is a felicitous consequence of typical manufacturing tolerances.     

Image SR-based NLM and DCNN improved IBP with cubic B-spline

ABSTRACT

Image super-resolution (SR) techniques aim to estimate high-resolution (HR) image from low-resolution (LR) image. Existing SR method has slow convergence and recovery of high-frequency details are inaccurate. To overcome these issues, two algorithms have been proposed for image SR based on non-local means improved iterative back projection (NLM-IIBP), deep convolutional neural network improved iterative back projection (DCNN-IIBP) to produce high-resolution images with low noise, minimal blur by restoring high-frequency details. In NLM-IIBP denoised images have been interpolated using cubic B-spline interpolation and processed using IIBP based on guided bilateral method. NLM preserves the edges effectively, but does not consider high dimensional information and over smoothing during noise minimization. To further improve the resolution, NLM is replaced by DCNN. DCNN denoising method suppresses different noises at different noise levels. The proposed algorithms have been analysed and compared with existing approaches using various parameters to prove the effectiveness.     

Harnessing defocus blur to recover high-resolution information in shape-from-focus technique

Traditional shape-from-focus (SFF) uses focus as the singular cue to derive the shape profile of a 3D object from a sequence of images. However, the stack of low-resolution (LR) observations is space-variantly blurred because of the finite depth of field of the camera. The authors propose to exploit the defocus information in the stack of LR images to obtain a super-resolved image as well as a high-resolution (HR) depth map of the underlying 3D object. Appropriate observation models are used to describe the image formation process in SFF. Local spatial dependencies of the intensities of pixels and their depth values are accounted for by modelling the HR image and the HR structure as independent Markov random fields. Taking as input the LR images from the stack and the LR depth map, the authors first obtain the super-resolved image of the 3D specimen and use it subsequently to reconstruct a HR depth profile of the object.     

Computer-controlled optical scanning tile microscope

A new type of computer-controlled optical scanning, high-magnification imaging system with a large field of view is described that overcomes the commonly believed incompatibility of achieving both high magnification and a large field of view. The new system incorporates galvanometer scanners, a CCD camera, and a high-brightness LED source for the fast acquisition of a large number of a high-resolution segmented tile images with a magnification of 800x for each tile. The captured segmented tile images are combined to create an effective enlarged view of a target totaling 1.6 mm x 1.2 mm in area. The speed and sensitivity of the system make it suitable for high-resolution imaging and monitoring of a small segmented area of 320 microm x 240 microm with 4 microm resolution. Each tile segment of the target can be zoomed up without loss of the high resolution. This new microscope imaging system gives both high magnification and a large field of view. This microscope can be utilized in medicine, biology, semiconductor inspection, device analysis, and quality control.     

Chemical imaging sensor and laser beacon

Design and functional aspects of PANSPEC, a panoramic-imaging chemical vapor sensor (PANSPEC is an abbreviation for infrared panoramic-viewing spectroradiometer), were advanced and its optical system reoptimized accordingly. The PANSPEC model unites camera and fused solid-state interferometer and photopolarimeter subsystems. The camera is an eye of the open atmosphere that collects, collimates, and images ambient infrared radiance from a panoramic field of view (FOV). The passive interferometer rapidly measures an infrared-absorbing (or infrared-emitting) chemical cloud traversing the FOV by means of molecular vibrational spectroscopy. The active photopolarimeter system provides a laser beam beacon. This beam carries identification (feature spectra measured by the interferometer) and heading (detector pixels disclosing these feature spectra) information on the hazardous cloud through a binary encryption of Mueller matrix elements. Interferometer and photopolarimeter share a common configuration of photoelastic modulation optics. PANSPEC was optimized for minimum aberrations and maximum resolution of image. The optimized design was evaluated for tolerances in the shaping and mounting of the optical system, stray light, and ghost images at the focal plane given a modulation transfer function metric.