Demosaicing using optimal recovery.

Color images in single-chip digital cameras are obtained by interpolating mosaiced color samples. These samples are encoded in a single-chip charge-coupled device by sampling the light after it passes through a color filter array (CFA) that contains different color filters (i.e., red, green, and blue) placed in some pattern. The resulting sparsely sampled images of the three-color planes are interpolated to obtain the complete color image. Interpolation usually introduces color artifacts due to the phase-shifted, aliased signals introduced by the sparse sampling of the CFAs. This paper introduces a nonlinear interpolation scheme based on edge information that produces high-quality visual results. The new method is especially good at reconstructing the image around edges, a place where the visual human system is most sensitive.     

Demosaicing Method for Digital Cameras with White‐RGB Color Filter Array

Demosaicing, or color filter array (CFA) interpolation, estimates missing color channels of raw mosaiced images from a CFA to reproduce full‐color images. It is an essential process for single‐sensor digital cameras with CFAs. In this paper, a new demosaicing method for digital cameras with Bayer‐like W‐RGB CFAs is proposed. To preserve the edge structure when reproducing full‐color images, we propose an edge direction–adaptive method using color difference estimation between different channels, which can be applied to practical digital camera use. To evaluate the performance of the proposed method in terms of CPSNR, FSIM, and S‐CIELAB color distance measures, we perform simulations on sets of mosaiced images captured by an actual prototype digital camera with a Bayer‐like W‐RGB CFA. The simulation results show that the proposed method demosaics better than a conventional one by approximately +22.4% CPSNR, +0.9% FSIM, and +36.7% S‐CIELAB distance.     

Reversing demosaicking and compression in color filter array image processing: performance analysis and modeling.

In the conventional processing chain of single-sensor digital still cameras (DSCs), the images are captured with color filter arrays (CFAs) and the CFA samples are demosaicked into a full color image before compression. To avoid additional data redundancy created by the demosaicking process, an alternative processing chain has been proposed to move the compression process before the demosaicking. Recent empirical studies have shown that the alternative chain can outperform the conventional one in terms of image quality at low compression ratios. To provide a theoretically sound basis for such conclusion, we propose analytical models for the reconstruction errors of the two processing chains. The models developed confirm the results of existing empirical studies and provide better understanding of DSC processing chains. The modeling also allows performance predictions for more advanced compression and demosaicking methods, thus providing important cues for future development in this area.     

Improvement of color video demosaicking in temporal domain.

Color demosaicking is critical to the image quality of digital still and video cameras that use a single-sensor array. Limited by the mosaic sampling pattern of the color filter array (CFA), color artifacts may occur in a demosaicked image in areas of high-frequency and/or sharp color transition structures. However, a color digital video camera captures a sequence of mosaic images and the temporal dimension of the color signals provides a rich source of information about the scene via camera and object motions. This paper proposes an inter-frame demosaicking approach to take advantage of all three forms of pixel correlations: spatial, spectral, and temporal. By motion estimation and statistical data fusion between adjacent mosaic frames, the new approach can remove much of the color artifacts that survive intra-frame demosaicking and also improve tone reproduction accuracy. Empirical results show that the proposed inter-frame demosaicking approach consistently outperforms its intra-frame counterparts both in peak signal-to-noise measure and subjective visual quality.     

Primary-consistent soft-decision color demosaicking for digital cameras (patent pending)

Color mosaic sampling schemes are widely used in digital cameras. Given the resolution of CCD sensor arrays, the image quality of digital cameras using mosaic sampling largely depends on the performance of the color demosaicking process. A common problem with existing color demosaicking algorithms is an inconsistency of sample interpolations in different primary color channels, which is the cause of the most objectionable color artifacts. To cure the problem, we propose a new primary-consistent soft-decision framework (PCSD) of color demosaicking. In the PCSD framework, we make multiple estimates of a missing color sample under different hypotheses on edge or texture directions. The estimates are made via a primary consistent interpolation, meaning that all three primary components of a color are interpolated in the same direction. The final estimate of a color sample is obtained by testing different interpolation hypotheses in the reconstructed full-resolution color image and selecting the best via an optimal statistical decision or inference process. A concrete color demosaicking method of the PCSD framework is presented. This new method eliminates certain types of color artifacts of existing color demosaicking methods. Extensive experimental results demonstrate that the PCSD approach can significantly improve the image quality of digital cameras in both subjective and objective measures. In some instances, our gain over the competing methods can be as much as 7 dB.     

PCA-Based Spatially Adaptive Denoising of CFA Images for Single-Sensor Digital Cameras

Single-sensor digital color cameras use a process called color demosaicking to produce full color images from the data captured by a color filter array (CFA). The quality of demosaicked images is degraded due to the sensor noise introduced during the image acquisition process. The conventional solution to combating CFA sensor noise is demosaicking first, followed by a separate denoising processing. This strategy will generate many noise-caused color artifacts in the demosaicking process, which are hard to remove in the denoising process. Few denoising schemes that work directly on the CFA images have been presented because of the difficulties arisen from the red, green and blue interlaced mosaic pattern, yet a well designed “denoising first and demosaicking later” scheme can have advantages such as less noise-caused color artifacts and cost-effective implementation. This paper presents a principle component analysis (PCA) based spatially-adaptive denoising algorithm, which works directly on the CFA data using a supporting window to analyze the local image statistics. By exploiting the spatial and spectral correlations existed in the CFA image, the proposed method can effectively suppress noise while preserving color edges and details. Experiments using both simulated and real CFA images indicate that the proposed scheme outperforms many existing approaches, including those sophisticated demosaicking and denoising schemes, in terms of both objective measurement and visual evaluation.      

A systems approach to custom VLSI for a digital color imagingsystem

An analysis of system requirements is presented that guides the design of a chip set that provides all the required functionality and control for single-sensor color imaging systems. The first device is a color filter array (CFA) processor that processes the single stream of sparse color information from an unconventional CFA pattern and produces a full-resolution color image in real time. The second chip is a red, green, and glue (RGB) processor that improves the image quality of the reconstructed RGB data by performing black-level adjustment, color matrixing, gamma correction, and edge enhancement, again in real time. The third device is a timing controller with an architecture specifically suited to imaging systems. The chips are algorithm specific, and the algorithms, architectures, and design methodology are detailed. The chip set is readily applicable to slide and negative film-to-video converters, electronic still cameras, and component or composite video cameras. It is capable of operation with NTSC, CCIR 601, and PAL/SECAM video standards     

Color reproduction from noisy CFA data of single sensor digital cameras.

Single sensor digital color still/video cameras capture images using a color filter array (CFA) and require color interpolation (demosaicking) to reconstruct full color images. The color reproduction has to combat sensor noises which are channel dependent. If untreated in demosaicking, sensor noises can cause color artifacts that are hard to remove later by a separate denoising process, because the demosaicking process complicates the noise characteristics by blending noises of different color channels. This paper presents a joint demosaicking-denoising approach to overcome this difficulty. The color image is restored from noisy mosaic data in two steps. First, the difference signals of color channels are estimated by linear minimum mean square-error estimation. This process exploits both spectral and spatial correlations to simultaneously suppress sensor noise and interpolation error. With the estimated difference signals, the full resolution green channel is recovered. The second step involves in a wavelet-based denoising process to remove the CFA channel-dependent noises from the reconstructed green channel. The red and blue channels are subsequently recovered. Simulated and real CFA mosaic data are used to evaluate the performance of the proposed joint demosaicking-denoising scheme and compare it with many recently developed sophisticated demosaicking and denoising schemes.     

Demosaicing algorithms for area- and line-scan cameras in print inspection

Most color image sensors use color filter arrays (CFA). With this sensor design the captured information at each sensor pixel position is restricted to a specific spectral portion (typically red, green and blue bands). To obtain the missing color responses at each pixel position, so-called CFA demosaicing algorithms are commonly used. We propose two new CFA demosaicing algorithms, which are well suited for industrial print inspection with respect to the requirements in accuracy and speed. As a main contribution, we introduce novel demosaicing algorithms for specific high-speed color digital time delay and integration (DTDI) CFA line-scan cameras. We compare the suggested CFA demosaicing algorithms to state-of-the art algorithms for area and line-scan camera operation modes. We show that the two new algorithms perform superior to conventional algorithms as indicated by reconstruction error.     

A low-complexity joint color demosaicking and zooming algorithm for digital camera.

This paper presents a low complexity joint color demosaicking and digital zooming algorithm for single-sensor digital cameras. The proposed algorithm directly extracts edge information from raw sensor data for interpolation in both demosaicking and zooming to preserve edge features in its output. This allows the extracted information to be exploited consistently in both stages and also efficiently, as no separate extraction process is required in different stages. The proposed algorithm can produce a zoomed full-color image as well as a zoomed Bayer color filter array image with outstanding performance as compared with conventional approaches which generally combine separate color demosaicking and digital zooming schemes.     

Adaptive filtering for color filter array demosaicking.

Most digital still cameras acquire imagery with a color filter array (CFA), sampling only one color value for each pixel and interpolating the other two color values afterwards. The interpolation process is commonly known as demosaicking. In general, a good demosaicking method should preserve the high-frequency information of imagery as much as possible, since such information is essential for image visual quality. We discuss in this paper two key observations for preserving high-frequency information in CFA demosaicking: (1) the high frequencies are similar across three color components, and (2) the high frequencies along the horizontal and vertical axes are essential for image quality. Our frequency analysis of CFA samples indicates that filtering a CFA image can better preserve high frequencies than filtering each color component separately. This motivates us to design an efficient filter for estimating the luminance at green pixels of the CFA image and devise an adaptive filtering approach to estimating the luminance at red and blue pixels. Experimental results on simulated CFA images, as well as raw CFA data, verify that the proposed method outperforms the existing state-of-the-art methods both visually and in terms of peak signal-to-noise ratio, at a notably lower computational cost.     

Novel color demosaicking for noisy color filter array data

Single sensor digital color still/video cameras use color demosaicking to reproduce full color images from color filter array (CFA) data. The quality of interpolated image will be degraded due to the sensor noise introduced during the image capture process. Many conventional demosaicking-denoising solutions adopt the channel-dependent noise model, which may fit the CMOS/CCD image sensor less than signal-dependent noise model. In this paper, the wavelet sub-band decomposition and synthesis are applied to interpolate the CFA data with signal-dependent noise model. The major contributions of this work include: (1) The combination of LMMSE and statistical calculation in wavelet domain are utilized to suppress the signal-dependent noise, which is separated into additive noise and multiplicative noise. (2) In CFA data, it has been verified that the quantitative relationship between the current pixel and the adjacent pixel, which locate in the same edge. Both simulated and real CFA images are employed to compare the proposed algorithm with the state-of-the-art techniques reported in the literature. The experimental results confirm that our method outperforms them both on demosaicking performance and on computational cost, when they process the noisy color filter array data.     

Color filter array demosaicking: new method and performance measures

Single-sensor digital cameras capture imagery by covering the sensor surface with a color filter array (CFA) such that each sensor pixel only samples one of three primary color values. To render a full-color image, an interpolation process, commonly referred to as CFA demosaicking, is required to estimate the other two missing color values at each pixel. In this paper, we present two contributions to the CFA demosaicking: a new and improved CFA demosaicking method for producing high quality color images and new image measures for quantifying the performance of demosaicking methods. The proposed demosaicking method consists of two successive steps: an interpolation step that estimates missing color values by exploiting spatial and spectral correlations among neighboring pixels, and a post-processing step that suppresses noticeable demosaicking artifacts by adaptive median filtering. Moreover, in recognition of the limitations of current image measures, we propose two types of image measures to quantify the performance of different demosaicking methods; the first type evaluates the fidelity of demosaicked images by computing the peak signal-to-noise ratio and CIELAB /spl utri/E/sup *//sub ab/ for edge and smooth regions separately, and the second type accounts for one major demosaicking artifact-zipper effect. We gauge the proposed demosaicking method and image measures using several existing methods as benchmarks, and demonstrate their efficacy using a variety of test images.     

Color plane interpolation using alternating projections

Most commercial digital cameras use color filter arrays to sample red, green, and blue colors according to a specific pattern. At the location of each pixel only one color sample is taken, and the values of the other colors must be interpolated using neighboring samples. This color plane interpolation is known as demosaicing; it is one of the important tasks in a digital camera pipeline. If demosaicing is not performed appropriately, images suffer from highly visible color artifacts. In this paper we present a new demosaicing technique that uses inter-channel correlation effectively in an alternating-projections scheme. We have compared this technique with six state-of-the-art demosaicing techniques, and it outperforms all of them, both visually and in terms of mean square error.     

A lossless compression scheme for Bayer color filter array images.

In most digital cameras, Bayer color filter array (CFA) images are captured and demosaicing is generally carried out before compression. Recently, it was found that compression-first schemes outperform the conventional demosaicing-first schemes in terms of output image quality. An efficient prediction-based lossless compression scheme for Bayer CFA images is proposed in this paper. It exploits a context matching technique to rank the neighboring pixels when predicting a pixel, an adaptive color difference estimation scheme to remove the color spectral redundancy when handling red and blue samples, and an adaptive codeword generation technique to adjust the divisor of Rice code for encoding the prediction residues. Simulation results show that the proposed compression scheme can achieve a better compression performance than conventional lossless CFA image coding schemes.     

Context-aware joint dictionary learning for color image demosaicking

Most digital cameras are overlaid with color filter arrays (CFA) on their electronic sensors, and thus only one particular color value would be captured at every pixel location. When producing the output image, one needs to recover the full color image from such incomplete color samples, and this process is known as demosaicking. In this paper, we propose a novel context-constrained demosaicking algorithm via sparse-representation based joint dictionary learning. Given a single mosaicked image with incomplete color samples, we perform color and texture constrained image segmentation and learn a dictionary with different context categories. A joint sparse representation is employed on different image components for predicting the missing color information in the resulting high-resolution image. During the dictionary learning and sparse coding processes, we advocate a locality constraint in our algorithm, which allows us to locate most relevant image data and thus achieve improved demosaicking performance. Experimental results show that the proposed method outperforms several existing or state-of-the-art techniques in terms of both subjective and objective evaluations.     

Robust Color Demosaicking With Adaptation to Varying Spectral Correlations

Almost all existing color demosaicking algorithms for digital cameras are designed on the assumption of high correlation between red, green, blue (or some other primary color) bands. They exploit spectral correlations between the primary color bands to interpolate the missing color samples, but in areas of no or weak spectral correlations, these algorithms are prone to large interpolation errors. Such demosaicking errors are visually objectionable because they tend to correlate with object boundaries and edges. This paper proposes a remedy to the above problem that has long been overlooked in the literature. The main contribution of this work is a hybrid demosaicking approach that supplements an existing color demosaicking algorithm by combining its results with those of adaptive intraband interpolation. This is formulated as an optimal data fusion problem, and two solutions are proposed: one is based on linear minimum mean-square estimation and the other based on support vector regression. Experimental results demonstrate that the new hybrid approach is more robust and eliminates the worst type of color artifacts of existing color demosaicking methods.      

基于小波的交替集合投射插值方法

大多数的数码相机都是根据特定的模板而使用颜色滤波阵列来对红、绿、蓝的颜色进行采样.每个像素的位置上只有一种颜色的采样,其他的颜色部分则需要插值恢复.这是数码相机的工作流程上一项重要的任务.该文提出了一种新的插值恢复技术,它有效利用了颜色信号之间相关性,采用基于小波的交替集合投射插值算法,这种算法可以实现更好的视觉效果,实验效果令人满意.     

The design and evaluation of a generic method for generating mosaicked multispectral filter arrays.

The technology of color filter arrays (CFA) has been widely used in the digital camera industry since it provides several advantages like low cost, exact registration, and strong robustness. The same motivations also drive the design of multispectral filter arrays (MSFA), in which more than three spectral bands are used. Although considerable research has been reported to optimally reconstruct the full-color image using various demosaicking algorithms, studies on the intrinsic properties of these filter arrays as well as the underlying design principles have been very limited. Given a set of representative spectral bands, the design of an MSFA involves two issues: the selection of tessellation mechanisms and the arrangement/layout of different spectral bands. We develop a generic MSFA generation method starting from a checkerboard pattern. We show, through case studies, that most of the CFAs currently used by the industry can be derived as special cases of MSFAs generated using the generic algorithm. The performance of different MSFAs are evaluated based on their intrinsic properties, namely, the spatial uniformity and the spectral consistency. We design two metrics, static coefficient and consistency coefficient, to measure these two parameters, respectively. The experimental results demonstrate that the generic algorithm can generate optimal or near-optimal MSFAs in both the rectangular and the hexagonal domains.     

Adaptive homogeneity-directed demosaicing algorithm.

A cost-effective digital camera uses a single-image sensor, applying alternating patterns of red, green, and blue color filters to each pixel location. A way to reconstruct a full three-color representation of color images by estimating the missing pixel components in each color plane is called a demosaicing algorithm. This paper presents three inherent problems often associated with demosaicing algorithms that incorporate two-dimensional (2-D) directional interpolation: misguidance color artifacts, interpolation color artifacts, and aliasing. The level of misguidance color artifacts present in two images can be compared using metric neighborhood modeling. The proposed demosaicing algorithm estimates missing pixels by interpolating in the direction with fewer color artifacts. The aliasing problem is addressed by applying filterbank techniques to 2-D directional interpolation. The interpolation artifacts are reduced using a nonlinear iterative procedure. Experimental results using digital images confirm the effectiveness of this approach.