Natural and Seamless Image Composition With Color Control

While the state-of-the-art image composition algorithms subtly handle the object boundary to achieve seamless image copy-and-paste, it is observed that they are unable to preserve the color fidelity of the source object, often require quite an amount of user interactions, and often fail to achieve realism when there exists salient discrepancy between the background textures in the source and destination images. These observations motivate our research towards color controlled natural and seamless image composition with least user interactions. In particular, based on the Poisson image editing framework, we first propose a variational model that considers both the gradient constraint and the color fidelity. The proposed model allows users to control the coloring effect caused by gradient domain fusion. Second, to have less user interactions, we propose a distance-enhanced random walks algorithm, through which we avoid the necessity of accurate image segmentation while still able to highlight the foreground object. Third, we propose a multiresolution framework to perform image compositions at different subbands so as to separate the texture and color components to simultaneously achieve smooth texture transition and desired color control. The experimental results demonstrate that our proposed framework achieves better and more realistic results for images with salient background color or texture differences, while providing comparable results as the state-of-the-art algorithms for images without the need of preserving the object color fidelity and without significant background texture discrepancy.     

Localizing text in scene images by boundary clustering, stroke segmentation, and string fragment classification

In this paper, we propose a novel framework to extract text regions from scene images with complex backgrounds and multiple text appearances. This framework consists of three main steps: boundary clustering (BC), stroke segmentation, and string fragment classification. In BC, we propose a new bigram-color-uniformity-based method to model both text and attachment surface, and cluster edge pixels based on color pairs and spatial positions into boundary layers. Then, stroke segmentation is performed at each boundary layer by color assignment to extract character candidates. We propose two algorithms to combine the structural analysis of text stroke with color assignment and filter out background interferences. Further, we design a robust string fragment classification based on Gabor-based text features. The features are obtained from feature maps of gradient, stroke distribution, and stroke width. The proposed framework of text localization is evaluated on scene images, born-digital images, broadcast video images, and images of handheld objects captured by blind persons. Experimental results on respective datasets demonstrate that the framework outperforms state-of-the-art localization algorithms.     

Color constancy by category correlation

Finding color representations that are stable to illuminant changes is still an open problem in computer vision. Until now, most approaches have been based on physical constraints or statistical assumptions derived from the scene, whereas very little attention has been paid to the effects that selected illuminants have on the final color image representation. The novelty of this paper is to propose perceptual constraints that are computed on the corrected images. We define the category hypothesis, which weights the set of feasible illuminants according to their ability to map the corrected image onto specific colors. Here, we choose these colors as the universal color categories related to basic linguistic terms, which have been psychophysically measured. These color categories encode natural color statistics, and their relevance across different cultures is indicated by the fact that they have received a common color name. From this category hypothesis, we propose a fast implementation that allows the sampling of a large set of illuminants. Experiments prove that our method rivals current state-of-art performance without the need for training algorithmic parameters. Additionally, the method can be used as a framework to insert top-down information from other sources, thus opening further research directions in solving for color constancy.     

Integrating color constancy into JPEG2000.

The human visual system is able to perceive colors as approximately constant. This ability is known as color constancy. In contrast, the colors measured by a sensor vary with the type of illuminant used. Color constancy is very important for digital photography and automatic color-based object recognition. In digital photography, this ability is known under the name automatic white balance. A number of algorithms have been developed for color constancy. We review two well-known color constancy algorithms, the gray world assumption and the Retinex algorithm and show how a color constancy algorithm may be integrated into the JPEG2000 framework. Since computer images are usually stored in compressed form anyway, little overhead is required to add color constancy into the processing pipeline.     

语义驱动的颜色恒常决策算法

不同颜色恒常性算法适用于不同场景下的图像,算法融合是扩展颜色恒常性算法适用范围常用的方法之一,而现有融合性算法在算法选择依据上忽略了语义信息在图像纹理特征描述中的作用,导致光源估计时的精度不高。针对该问题,提出一种语义驱动的颜色恒常决策算法。首先,利用PSPNet(Pyramid Scene Parsing Network)模型对经过一阶灰度边缘算法(1st Gray Edge)偏色预处理后的目标图像进行场景语义分割,并计算场景中各个语义类别的占比;其次,根据语义类别及占比在已训练的决策集合中寻找相似的参考图像,并使用欧氏距离计算两者的语义相似度;最后,将语义相似度与基于多维欧氏空间确定的阈值进行判别,根据判别结果选择合适算法为目标图像实行偏色校正。在Color Checker和NUS-8 camera两种数据集中的实验结果表明,所提算法光源估计角度误差较单一算法均大幅度下降,且较同类型融合性算法分别下降14.02%和8.17%,提高了光源估计的鲁棒性和准确度。     

Multiple transformation function estimation for image enhancement

Most deep learning-based image enhancement algorithms have been developed based on the image-to-image translation approach, in which enhancement processes are difficult to interpret. In this paper, we propose a novel interpretable image enhancement algorithm that estimates multiple transformation functions to describe complex color mapping. First, we develop a histogram-based multiple transformation function estimation network (HMTF-Net) to estimate multiple transformation functions by exploiting both the spatial and statistical information of the input images. Second, we estimate pixel-wise weight maps, which indicate the contribution of each transformation function at each pixel, based on the local structures of the input image and the transformed images obtained by each transformation function. Finally, we obtain the enhanced image as the weighted sum of the transformed images using the estimated weight maps. Extensive experiments confirm the effectiveness of the proposed approach and demonstrate that the proposed algorithm outperforms state-of-the-art image enhancement algorithms for different image enhancement tasks.     

城市航空影像中基于颜色恒常性的阴影消除

基于颜色恒常性理论,对真彩色和彩红外城市航空影像中高大建筑物形成的阴影进行消除。首先采用光谱比技术和最大类间方差法(Otsu)阈值分割技术进行城市航空影像中建筑物阴影的检测,进而就颜色恒常计算的Shades of Gray算法中明可夫斯基范式(Minkowski norm)的p取不同值情况下的阴影去除效果进行实验,利用亮度、对比度及平均梯度值比较阴影去除效果的好坏。实验表明:在基于航空影像阴影区域及非阴影区域划分的基础上,本文方法比一般的阴影区反差拉伸方法效果好;且与一般场景影像的阴影去除不同,对两类航空影像,p取2时阴影去除效果最佳,说明这两类影像不能简单看成是一个灰色世界影像。     

A fast white balance algorithm based on pixel greyness

The goal of automatic white balance (AWB) is to maintain colour constancy of an image by removing colour cast caused by un-canonical illuminant. In this paper, we address two limitations associated with a class of AWB algorithms and propose a technique to estimate the illuminant which takes into consideration the internal illumination and all pixels of the image. The estimate is calculated by a weighted average of all pixels. The weight for a pixel is determined from the greyness. The greyness of a pixel is measured from its chroma Cb and Cr in the YCbCr colour space. The experimental results demonstrate that performance of the proposed technique is competitive with that of state-of-the-art AWB algorithms. The proposed algorithm can be implemented in real-time applications, such as in consumer digital cameras due to its low computational complexity.     

一种基于局部可信视差的立体图像误码掩盖算法

为解决立体图像传输的差错问题,提出了一种基于局部可信视差的掩盖方法。首先,考虑到存在丢失块,并结合像素间色彩空间相似程度和几何空间距离接近程度等因素,设计了基准点偏置的窗口用于自适应权重的视差匹配;然后根据视差连续性原则和左右一致性约束,得出局部的可信视差;最后采用"胜者为王"(Winner-Takes-All)策略估计丢失块的视差,并根据此视差提取相应块来进行误码掩盖.实验结果表明,与其他掩盖算法相比,该算法在计算复杂度相当的情况下,无论在重建图像的PSNR还是主观质量上均具有较好的效果.     

A new class of chromatic filters for color image processing. theory and applications

This paper advances a new framework for chromatic filtering of color images. The chromatic content of a color image is encoded in the CIE u'v' chromaticity coordinates whereas the achromatic content is encoded as CIE Y tristimulus value. Within the u'v' chromaticity diagram, colors are added according to the well-known center of gravity law of additive color mixtures, which is generalized here into a nonlinear filtering scheme for processing the two chromatic signals u' and v'. The achromatic channel Y can be processed with traditional filtering schemes, either linear or nonlinear, depending on the specific task at hand. The most interesting characteristics of the new filtering scheme are: 1) the elimination of color smearing effects along edges between bright and dark areas; 2) the possibility of processing chromatic components in a noniterative fashion through linear convolution operations; and 3) the consequent amenability to computationally efficient implementations with fast Fourier transform. The paper includes several examples with both synthetic and real images where the performance of the new filtering method is compared with that of other color image processing algorithms.     

Graph-based semi-supervised learning with multiple labels

Conventional graph-based semi-supervised learning methods predominantly focus on single label problem. However, it is more popular in real-world applications that an example is associated with multiple labels simultaneously. In this paper, we propose a novel graph-based learning framework in the setting of semi-supervised learning with multiple labels. This framework is characterized by simultaneously exploiting the inherent correlations among multiple labels and the label consistency over the graph. Based on the proposed framework, we further develop two novel graph-based algorithms. We apply the proposed methods to video concept detection over TRECVID 2006 corpus and report superior performance compared to the state-of-the-art graph-based approaches and the representative semi-supervised multi-label learning methods.     

Stereo matching using cost volume watershed and region merging

Segment based disparity estimation methods have been proposed in many different ways. Most of these studies are built upon the hypothesis that no large disparity jump exists within a segment. When this hypothesis does not hold, it is difficult for these methods to estimate disparities correctly. Therefore, these methods work well only when the images are initially over segmented but do not work well for under segmented cases. To solve this problem, we present a new segment based stereo matching method which consists of two algorithms: a cost volume watershed algorithm (CVW) and a region merging (RM) algorithm. For incorrectly under segmented regions where pixels on different objects are grouped into one segment, the CVW algorithm regroups the pixels on different objects into different segments and provides disparity estimation to the pixels in different segments accordingly. For unreliable and occluded regions, we merge them into neighboring reliable segments for robust disparity estimation. The comparison between our method and the current state-of-the-art methods shows that our method is very competitive and is robust particularly when the images are initially under segmented.     

基于全变分Retinex及梯度域雾天图像增强算法

为提高雾天图像增强的对比度并保持颜色恒常性,提出了基于全变分Retinex及梯度域雾天图像增强算法。首先,采用高斯—赛德尔GS（Gauss-Seidel）迭代算法对基于Retinex的全变分能量泛函数进行求解,从而有效地保持颜色恒常性;其次,采用相对梯度与绝对梯度相结合的方式拉伸雾天图像较亮处的梯度, 在全变分Retinex理论下重建增强后的雾天图像,并将该增强算法应用到彩色图像;最后,加权融合基于全变分Retinex增强算法与梯度域增强算法的增强结果,使得增强结果既能提高对比度又能保持色彩恒常性。实验结果表明,本算法提高了雾天图像增强后的对比度和清晰度,具有颜色恒常性、颜色保真高等特性。     

Enhancing underwater image via color correction and Bi-interval contrast enhancement

Underwater captured images often suffer from color cast and low visibility due to light is scattered and absorbed while it traveling in water. In this paper, we proposed a novel method of color correction and Bi-interval contrast enhancement to improve the quality of underwater images. Firstly, a simple and effective color correction method based on sub-interval linear transformation is employed to address color distortion. Then, a Gaussian low-pass filter is applied to the L channel to decompose the low- and high-frequency components. Finally, the low- and high-frequency components are enhanced by Bi-interval histogram based on optimal equalization threshold strategy and S-shaped function to enhancement image contrast and highlight image details. Inspired by the multi-scale fusion, we employed a simple linear fusion to integrate the enhanced high- and low-frequency components. Comparison with state-of-the-art methods show that the proposed method outputs high-quality underwater images with qualitative and quantitative evaluation well.     

直方图平移的色彩还原算法研究

本文对应用于AMOLED的低功耗色彩还原算法开展研究,提出了一种基于直方图平移的算法。通过直方图平移增大直方图重叠面积,从而得到优良的色彩还原效果。由于采用居中的直方图作为平移基准,在保证色彩还原性能的同时降低了显示屏端功耗。与传统的算法相比,该算法只有加减运算,使得硬件实现的复杂度大大降低。经验证,该算法具有优良的性能表现,其直方图重叠面积(OA)比灰度世界算法高23%,比白点补丁算法高43%。此外,该算法处理所得图像在屏端的显示功耗(PD)是灰度世界算法的96%,是白点补丁算法的73%。     

EGAN: Non-uniform image deblurring based on edge adversarial mechanism and partial weight sharing network

In this paper, we propose a solution to transform spatially variant blurry images into the photo-realistic sharp manifold. Image deblurring task is valuable and challenging in computer vision. However, existing learning-based methods cannot produce images with clear edges and fine details, which exhibit significant challenges for generated-based loss functions used in existing methods. Instead of only designing architectures and loss functions for generators, we propose a generative adversarial network (GAN) framework based on an edge adversarial mechanism and a partial weight sharing network. In order to propel the entire network to learn image edges information consciously, we propose an edge reconstruction loss function and an edge adversarial loss function to restrict the generator and the discriminator respectively. We further introduce a partial weight sharing structure, the sharp features from clean images encourage the recovery of image details of deblurred images. The proposed partial weight sharing structure improves image details effectively. Experimental results show that our method is able to generate photo-realistic sharp images from real-world blurring images and outperforms state-of-the-art methods.     

Multi-scale and multi-patch transformer for sandstorm image enhancement

Sandstorm is a meteorological phenomenon common in arid and semi-arid regions. A sandstorm can carry large volumes of sand unexpectedly, which leads to severe color deviations and significantly degraded visibility when an image is taken in such a scenario. However, existing image enhancement methods cannot enhance sandstorm images well due to the challenging degradations and the scarcity of sandstorm training data. In this paper, we propose a Transformer with rotary position embedding to perform sandstorm image enhancement via building multi-scale and multi-patch dependencies. Our key insights in this work are 1) a multi-scale Transformer can globally eliminate the color deviations of sandstorm images via aggregating global information, 2) a multi-patch Transformer can recover local details well via learning the spatial variant degradations, and 3) a U-shape Transformer with rotary position embedding as the core unit of multi-scale and multi-patch Transformer can effectively build the long-range dependencies. We also contribute a real-world Sandstorm Image Enhancement (SIE) dataset including 1,400 sandstorm images with different degrees of degradations and various scenes. Experiments performed on synthetic images and real-world sandstorm images demonstrate that our proposed method not only obtains visually pleasing results but also outperforms state-of-the-art methods qualitatively and quantitatively.     

A wrapper-based approach to image segmentation and classification.

The traditional processing flow of segmentation followed by classification in computer vision assumes that the segmentation is able to successfully extract the object of interest from the background image. It is extremely difficult to obtain a reliable segmentation without any prior knowledge about the object that is being extracted from the scene. This is further complicated by the lack of any clearly defined metrics for evaluating the quality of segmentation or for comparing segmentation algorithms. We propose a method of segmentation that addresses both of these issues, by using the object classification subsystem as an integral part of the segmentation. This will provide contextual information regarding the objects to be segmented, as well as allow us to use the probability of correct classification as a metric to determine the quality of the segmentation. We view traditional segmentation as a filter operating on the image that is independent of the classifier, much like the filter methods for feature selection. We propose a new paradigm for segmentation and classification that follows the wrapper methods of feature selection. Our method wraps the segmentation and classification together, and uses the classification accuracy as the metric to determine the best segmentation. By using shape as the classification feature, we are able to develop a segmentation algorithm that relaxes the requirement that the object of interest to be segmented must be homogeneous in some low-level image parameter, such as texture, color, or grayscale. This represents an improvement over other segmentation methods that have used classification information only to modify the segmenter parameters, since these algorithms still require an underlying homogeneity in some parameter space. Rather than considering our method as, yet, another segmentation algorithm, we propose that our wrapper method can be considered as an image segmentation framework, within which existing image segmentation algorithms may be executed. We show the performance of our proposed wrapper-based segmenter on real-world and complex images of automotive vehicle occupants for the purpose of recognizing infants on the passenger seat and disabling the vehicle airbag. This is an interesting application for testing the robustness of our approach, due to the complexity of the images, and, consequently, we believe the algorithm will be suitable for many other real-world applications.