Color filter array demosaicking: new method and performance measures

Single-sensor digital cameras capture imagery by covering the sensor surface with a color filter array (CFA) such that each sensor pixel only samples one of three primary color values. To render a full-color image, an interpolation process, commonly referred to as CFA demosaicking, is required to estimate the other two missing color values at each pixel. In this paper, we present two contributions to the CFA demosaicking: a new and improved CFA demosaicking method for producing high quality color images and new image measures for quantifying the performance of demosaicking methods. The proposed demosaicking method consists of two successive steps: an interpolation step that estimates missing color values by exploiting spatial and spectral correlations among neighboring pixels, and a post-processing step that suppresses noticeable demosaicking artifacts by adaptive median filtering. Moreover, in recognition of the limitations of current image measures, we propose two types of image measures to quantify the performance of different demosaicking methods; the first type evaluates the fidelity of demosaicked images by computing the peak signal-to-noise ratio and CIELAB /spl utri/E/sup *//sub ab/ for edge and smooth regions separately, and the second type accounts for one major demosaicking artifact-zipper effect. We gauge the proposed demosaicking method and image measures using several existing methods as benchmarks, and demonstrate their efficacy using a variety of test images.     

Improvement of color video demosaicking in temporal domain.

Color demosaicking is critical to the image quality of digital still and video cameras that use a single-sensor array. Limited by the mosaic sampling pattern of the color filter array (CFA), color artifacts may occur in a demosaicked image in areas of high-frequency and/or sharp color transition structures. However, a color digital video camera captures a sequence of mosaic images and the temporal dimension of the color signals provides a rich source of information about the scene via camera and object motions. This paper proposes an inter-frame demosaicking approach to take advantage of all three forms of pixel correlations: spatial, spectral, and temporal. By motion estimation and statistical data fusion between adjacent mosaic frames, the new approach can remove much of the color artifacts that survive intra-frame demosaicking and also improve tone reproduction accuracy. Empirical results show that the proposed inter-frame demosaicking approach consistently outperforms its intra-frame counterparts both in peak signal-to-noise measure and subjective visual quality.     

Colour demosaicking method using adaptive cubic convolution interpolation with sequential averaging

Colour demosaicking refers to the process used to estimate missing colour components at each pixel position in a single-sensor imaging device equipped with a colour filter array. The proposed method starts the demosaicking process by estimating the luminance channel, which utilises cubic convolution interpolation along the direction of the smallest gradient magnitude at each chrominance pixel position. Reconstruction of chrominance channels comprises spectral correlation-based average of neighbouring chrominance pixels and sequential filtering on the reconstructed chrominance channels. The sequential filtering can be regarded as a postprocessing stage and effectively smears out colour artefacts. Experimental results show the effectiveness of this approach     

PCA-Based Spatially Adaptive Denoising of CFA Images for Single-Sensor Digital Cameras

Single-sensor digital color cameras use a process called color demosaicking to produce full color images from the data captured by a color filter array (CFA). The quality of demosaicked images is degraded due to the sensor noise introduced during the image acquisition process. The conventional solution to combating CFA sensor noise is demosaicking first, followed by a separate denoising processing. This strategy will generate many noise-caused color artifacts in the demosaicking process, which are hard to remove in the denoising process. Few denoising schemes that work directly on the CFA images have been presented because of the difficulties arisen from the red, green and blue interlaced mosaic pattern, yet a well designed “denoising first and demosaicking later” scheme can have advantages such as less noise-caused color artifacts and cost-effective implementation. This paper presents a principle component analysis (PCA) based spatially-adaptive denoising algorithm, which works directly on the CFA data using a supporting window to analyze the local image statistics. By exploiting the spatial and spectral correlations existed in the CFA image, the proposed method can effectively suppress noise while preserving color edges and details. Experiments using both simulated and real CFA images indicate that the proposed scheme outperforms many existing approaches, including those sophisticated demosaicking and denoising schemes, in terms of both objective measurement and visual evaluation.      

A low-complexity joint color demosaicking and zooming algorithm for digital camera.

This paper presents a low complexity joint color demosaicking and digital zooming algorithm for single-sensor digital cameras. The proposed algorithm directly extracts edge information from raw sensor data for interpolation in both demosaicking and zooming to preserve edge features in its output. This allows the extracted information to be exploited consistently in both stages and also efficiently, as no separate extraction process is required in different stages. The proposed algorithm can produce a zoomed full-color image as well as a zoomed Bayer color filter array image with outstanding performance as compared with conventional approaches which generally combine separate color demosaicking and digital zooming schemes.     

基于压缩感知的正六边形CFA模式彩色图像去马赛克方法

针对基于四边形排列的去马赛克(Demosaicking)的 传统方法存在拉链现象和虚假色等问题,本文尝试将更加符合人眼视觉特性的六边形采 样方式应用于彩色图像成像 系统,并从图像稀疏特性角度出发,提出基于压缩感知(Compressive sensing,CS)框架的 彩色图像去马赛克方法。本文方法 充分挖掘了彩色分量间和分量内的稀疏特性,可使复原图像的纹理细节与色彩更加逼真,有 效地避免了拉链现象和虚假色现象。实验结果验证了本文方法的有效性。     

Context-aware joint dictionary learning for color image demosaicking

Most digital cameras are overlaid with color filter arrays (CFA) on their electronic sensors, and thus only one particular color value would be captured at every pixel location. When producing the output image, one needs to recover the full color image from such incomplete color samples, and this process is known as demosaicking. In this paper, we propose a novel context-constrained demosaicking algorithm via sparse-representation based joint dictionary learning. Given a single mosaicked image with incomplete color samples, we perform color and texture constrained image segmentation and learn a dictionary with different context categories. A joint sparse representation is employed on different image components for predicting the missing color information in the resulting high-resolution image. During the dictionary learning and sparse coding processes, we advocate a locality constraint in our algorithm, which allows us to locate most relevant image data and thus achieve improved demosaicking performance. Experimental results show that the proposed method outperforms several existing or state-of-the-art techniques in terms of both subjective and objective evaluations.     

Novel color demosaicking for noisy color filter array data

Single sensor digital color still/video cameras use color demosaicking to reproduce full color images from color filter array (CFA) data. The quality of interpolated image will be degraded due to the sensor noise introduced during the image capture process. Many conventional demosaicking-denoising solutions adopt the channel-dependent noise model, which may fit the CMOS/CCD image sensor less than signal-dependent noise model. In this paper, the wavelet sub-band decomposition and synthesis are applied to interpolate the CFA data with signal-dependent noise model. The major contributions of this work include: (1) The combination of LMMSE and statistical calculation in wavelet domain are utilized to suppress the signal-dependent noise, which is separated into additive noise and multiplicative noise. (2) In CFA data, it has been verified that the quantitative relationship between the current pixel and the adjacent pixel, which locate in the same edge. Both simulated and real CFA images are employed to compare the proposed algorithm with the state-of-the-art techniques reported in the literature. The experimental results confirm that our method outperforms them both on demosaicking performance and on computational cost, when they process the noisy color filter array data.     

A Color CMOS Imager With 4 $times$ 4 White-RGB Color Filter Array for Increased Low-Illumination Signal-to-Noise Ratio

A color CMOS image sensor with the 4 times 4 White-RGB color filter array (CFA) including 50% white pixels has been developed. A transparent layer has been fabricated on the white pixel to realize over 95% transmission for visible light with wavelengths of 400-700 nm. Pixel pitch and number of the pixels were 3.3 mum and 2 million, respectively. With the simple and low-noise color separation process, low-illumination signal-to-noise ratios of luminance signal have been increased by 6 dB, compared with those of the Bayer pattern. Moreover, by locating the pixels so that every color components can be detected in every column and line, color artifacts at the edge were suppressed. The edge detection process became unnecessary and the process time was reduced by 70%. The new CFA has the potential to significantly increase the sensitivity of CMOS/CCD image sensors.     

引入双边滤波器优化的彩色滤波阵列插值

基于彩色滤波阵列(CFA)的图像传感器在每个像素 位置获得三原色红(R)、绿(G)和蓝(B)中的一种分量 ,其缺失 分量需要根据周围像素插值得到。目前提出的许多种插值算法,绝大部分采用Bayer排列模 式。本文在色 差恒定假设基础上,提出一种基于双边滤波器的自适应Bayer模型插值算法,对G通道的估计采 用自适应滤波器进行插值,对R和B通道的插值采用双边滤波器。算法利用待插值像素 与不同距离像 素相关性不同的思想,根据图像边缘自适应设定滤波模板,能较准确估计G、B和R、G通 道之间的色差值。 实验结果表明,对比多尺度色差梯度算法和边缘强度滤波等算法,插值后的图像不仅 主观视觉, 且客观评价指标(彩色峰值信噪比,CPSNR)均优于这些算法。     

Spatio-spectral color filter array design for optimal image recovery

In digital imaging applications, data are typically obtained via a spatial subsampling procedure implemented as a color filter array-a physical construction whereby only a single color value is measured at each pixel location. Owing to the growing ubiquity of color imaging and display devices, much recent work has focused on the implications of such arrays for subsequent digital processing, including in particular the canonical demosaicking task of reconstructing a full color image from spatially subsampled and incomplete color data acquired under a particular choice of array pattern. In contrast to the majority of the demosaicking literature, we consider here the problem of color filter array design and its implications for spatial reconstruction quality. We pose this problem formally as one of simultaneously maximizing the spectral radii of luminance and chrominance channels subject to perfect reconstruction, and-after proving sub-optimality of a wide class of existing array patterns-provide a constructive method for its solution that yields robust, new panchromatic designs implementable as subtractive colors. Empirical evaluations on multiple color image test sets support our theoretical results, and indicate the potential of these patterns to increase spatial resolution for fixed sensor size, and to contribute to improved reconstruction fidelity as well as significantly reduced hardware complexity.     

Reversing demosaicking and compression in color filter array image processing: performance analysis and modeling.

In the conventional processing chain of single-sensor digital still cameras (DSCs), the images are captured with color filter arrays (CFAs) and the CFA samples are demosaicked into a full color image before compression. To avoid additional data redundancy created by the demosaicking process, an alternative processing chain has been proposed to move the compression process before the demosaicking. Recent empirical studies have shown that the alternative chain can outperform the conventional one in terms of image quality at low compression ratios. To provide a theoretically sound basis for such conclusion, we propose analytical models for the reconstruction errors of the two processing chains. The models developed confirm the results of existing empirical studies and provide better understanding of DSC processing chains. The modeling also allows performance predictions for more advanced compression and demosaicking methods, thus providing important cues for future development in this area.     

Color reproduction from noisy CFA data of single sensor digital cameras.

Single sensor digital color still/video cameras capture images using a color filter array (CFA) and require color interpolation (demosaicking) to reconstruct full color images. The color reproduction has to combat sensor noises which are channel dependent. If untreated in demosaicking, sensor noises can cause color artifacts that are hard to remove later by a separate denoising process, because the demosaicking process complicates the noise characteristics by blending noises of different color channels. This paper presents a joint demosaicking-denoising approach to overcome this difficulty. The color image is restored from noisy mosaic data in two steps. First, the difference signals of color channels are estimated by linear minimum mean square-error estimation. This process exploits both spectral and spatial correlations to simultaneously suppress sensor noise and interpolation error. With the estimated difference signals, the full resolution green channel is recovered. The second step involves in a wavelet-based denoising process to remove the CFA channel-dependent noises from the reconstructed green channel. The red and blue channels are subsequently recovered. Simulated and real CFA mosaic data are used to evaluate the performance of the proposed joint demosaicking-denoising scheme and compare it with many recently developed sophisticated demosaicking and denoising schemes.     

Binary tree-based generic demosaicking algorithm for multispectral filter arrays.

In this paper, we extend the idea of using mosaicked color filter array (CFA) in color imaging, which has been widely adopted in the digital color camera industry, to the use of multispectral filter array (MSFA) in multispectral imaging. The filter array technique can help reduce the cost, achieve exact registration, and improve the robustness of the imaging system. However, the extension from CFA to MSFA is not straightforward. First, most CFAs only deal with a few bands (3 or 4) within the narrow visual spectral region, while the design of MSFA needs to handle the arrangement of multiple bands (more than 3) across a much wider spectral range. Second, most existing CFA demosaicking algorithms assume the fixed Bayer CFA and are confined to properties only existed in the color domain. Therefore, they cannot be directly applied to multispectral demosaicking. The main challenges faced in multispectral demosaicking is how to design a generic algorithm that can handle the more diversified MSFA patterns, and how to improve performance with a coarser spatial resolution and a less degree of spectral correlation. In this paper, we present a binary tree based generic demosaicking method. Two metrics are used to evaluate the generic algorithm, including the root mean-square error (RMSE) for reconstruction performance and the classification accuracy for target discrimination performance. Experimental results show that the demosaicked images present low RMSE (less than 7) and comparable classification performance as original images. These results support that MSFA technique can be applied to multispectral imaging with unique advantages.     

Color demosaicking via directional linear minimum mean square-error estimation.

Digital cameras sample scenes using a color filter array of mosaic pattern (e.g., the Bayer pattern). The demosaicking of the color samples is critical to the image quality. This paper presents a new color demosaicking technique of optimal directional filtering of the green-red and green-blue difference signals. Under the assumption that the primary difference signals (PDS) between the green and red/blue channels are low pass, the missing green samples are adaptively estimated in both horizontal and vertical directions by the linear minimum mean square-error estimation (LMMSE) technique. These directional estimates are then optimally fused to further improve the green estimates. Finally, guided by the demosaicked full-resolution green channel, the other two color channels are reconstructed from the LMMSE filtered and fused PDS. The experimental results show that the presented color demosaicking technique outperforms the existing methods both in PSNR measure and visual perception.     

基于同态滤波的彩色图像光照补偿方法

为了消除光照对彩色图像的影响,提出了一种基于同态滤波的光照补偿方法。在频域内采用同态滤波对图像进行处理,然后将巴特沃斯高通滤波传递函数引入同态滤波器中,设计出一种新的动态巴特沃斯同态滤波器,增强图像在YIQ色彩空间中的亮度分量,并保持图像色度分量不变。在增加图像高频分量的同时,削减低频分量,弥补由光照不足引起的图像质量下降,最终实现对图像的光照补偿。实验结果表明,该方法在弱化光照影响,提高图像质量方面具有良好的性能,是一种比传统高斯高通滤波和直方图均衡更有效的彩色图像光照补偿方法。     

非局部相似性去马赛克算法

单传感器数码相机得到的色彩图像在每一个像素点处只有一种色彩值,为了得到一幅全彩色图像,需要在每一个像素位置上估计出另外两个缺失的色彩值。现有主要算法都是利用像素的相关性进行估计和插值,在那些边缘色彩跳变处和色彩高饱和度处容易估计失误,出现所谓的马赛克失真。为了克服这类马赛克现象,本文提出了一种利用图像的非局部相似性,即利用处于图像中不同位置处的像素点往往表现出很强的相关性这一特点,结合图像内容的局部平坦度自适应去马赛克的插值算法。该算法,首先根据相似度函数搜索与被插像素最相似的像素,然后利用区域水平和垂直方向的梯度组算子来计算区域的平坦度,从而根据相似程度和平坦度自适应地选择图像块进行插值。实验结果表明,相对于传统插值算法,该算法提高了图像的峰值信噪比,锐化了图像的纹理和边缘,减少了虚假色和锯齿现象,改善了图像的视觉效果。      

基于像素滤波和中值滤波的深度图像修复方法

针对Kinect相机采集的深度数据一般有噪声和黑 洞现象,直接应用于场景三维重建系统中效果差的问 题,提出了像素滤波器和中值滤波器相结合的Kinect 深度图像修复方法。首先对一帧图片 上的像素进行搜索, 找到像素值为0的点,以该点为中心,利用其邻域内的像素,定义一个两层的滤波器,根据 像素滤波器的原理 对其进行修复,填充深度图片的黑洞;然后采用中值滤波器,在平滑深度图像的同时保留边 缘信息,去除孤立 噪声点。实验结果表明,本文方法去噪的同时也能对黑洞进行修补,与原始深度图像相比, 空洞明显减少,深度图像质量大大提高。     

基于NSST域混合滤波与改进边缘检测PCNN的医学图像融合

针对传统医学图像融合中存在细节模糊、能量保存不完整、运行时间长等问题,提出一种基于非下采样剪切波(non-subsampled shearlet transform, NSST)域混合滤波与改进边缘检测脉冲耦合神经网络(pulse coupled neural network, PCNN)的医学图像融合方法。首先,利用YUV模型进行颜色空间转换分离出亮度通道Y,接着利用混合滤波分别对源核磁共振(magnetic resonance imaging, MRI)图像和亮度通道的灰度图像进行不同程度的增强。其次,采用NSST对增强后的MRI和亮度通道的灰度图像进行分解,得到高低频子带。低频子带使用修正的拉普拉斯能量和(weighted sum of eight-neighborhood-based modified Laplacian,WSEML)与局部区域能量加权和(weight local energy,WLE)的融合策略,高频子带采用改进边缘检测PCNN的融合策略。最后,经NSST逆变换得到融合图像。通过与其他6种融合方法对比,本文方法可以有效提高图像融合过程中的细节提取和能量保存,且...     

Secure single-sensor digital camera

An efficient secure single-sensor camera for captured image copyright protection and ownership authentication is introduced. Visible watermarks are embedded during the colour filter array (CFA) data acquisition process. Subsequent demosaicking of the watermarked CFA grey-scale images generates secure full-colour images readily available for storage or distribution.