无监督深度学习模型的多聚焦图像融合算法

深度学习技术应用到多聚焦图像融合领域时,其大多通过监督学习的方式来训练网络,但由于缺乏专用于多聚焦图像融合的监督训练的标记数据集,且制作专用的大规模标记训练集代价过高,所以现有方法多通过在聚焦图像中随机添加高斯模糊进行监督学习,这导致网络训练难度大,很难实现理想的融合效果。为解决以上问题,提出了一种易实现且融合效果好的多聚焦图像融合方法。通过在易获取的无标记数据集上以无监督学习方式训练引入了注意力机制的encoder-decoder网络模型,获得输入源图像的深层特征。再通过形态聚焦检测对获取的特征进行活动水平测量生成初始决策图。运用一致性验证方法对初始决策图优化,得到最终的决策图。融合图像质量在主观视觉和客观指标两方面上进行评定,经实验结果表明,融合图像清晰度高,保有细节丰富且失真度小。     

Multi-focus image fusion using Content Adaptive Blurring

Multi-focus image fusion has emerged as an important research area in information fusion. It aims at increasing the depth-of-field by extracting focused regions from multiple partially focused images, and merging them together to produce a composite image in which all objects are in focus. In this paper, a novel multi-focus image fusion algorithm is presented in which the task of detecting the focused regions is achieved using a Content Adaptive Blurring (CAB) algorithm. The proposed algorithm induces non-uniform blur in a multi-focus image depending on its underlying content. In particular, it analyzes the local image quality in a neighborhood and determines if the blur should be induced or not without losing image quality. In CAB, pixels belonging to the blur regions receive little or no blur at all, whereas the focused regions receive significant blur. Absolute difference of the original image and the CAB-blurred image yields initial segmentation map, which is further refined using morphological operators and graph-cut techniques to improve the segmentation accuracy. Quantitative and qualitative evaluations and comparisons with current state-of-the-art on two publicly available datasets demonstrate the strength of the proposed algorithm.     

Multi-scale convolutional neural network for multi-focus image fusion

In this study, we present new deep learning (DL) method for fusing multi-focus images. Current multi-focus image fusion (MFIF) approaches based on DL methods mainly treat MFIF as a classification task. These methods use a convolutional neural network (CNN) as a classifier to identify pixels as focused or defocused pixels. However, due to unavailability of labeled data to train networks, existing DL-based supervised models for MFIF add Gaussian blur in focused images to produce training data. DL-based unsupervised models are also too simple and only applicable to perform fusion tasks other than MFIF. To address the above issues, we proposed a new MFIF method, which aims to learn feature extraction, fusion and reconstruction components together to produce a complete unsupervised end-to-end trainable deep CNN. To enhance the feature extraction capability of CNN, we introduce a Siamese multi-scale feature extraction module to achieve a promising performance. In our proposed network we applied multiscale convolutions along with skip connections to extract more useful common features from a multi-focus image pair. Instead of using basic loss functions to train the CNN, our model utilizes structure similarity (SSIM) measure as a training loss function. Moreover, the fused images are reconstructed in a multiscale manner to guarantee more accurate restoration of images. Our proposed model can process images with variable size during testing and validation. Experimental results on various test images validate that our proposed method yields better quality fused images that are superior to the fused images generated by compared state-of-the-art image fusion methods.     

自学习规则下的多聚焦图像融合

目的 基于深度学习的多聚焦图像融合方法主要是利用卷积神经网络（convolutional neural network,CNN）将像素分类为聚焦与散焦。监督学习过程常使用人造数据集,标签数据的精确度直接影响了分类精确度,从而影响后续手工设计融合规则的准确度与全聚焦图像的融合效果。为了使融合网络可以自适应地调整融合规则,提出了一种基于自学习融合规则的多聚焦图像融合算法。方法 采用自编码网络架构,提取特征,同时学习融合规则和重构规则,以实现无监督的端到端融合网络;将多聚焦图像的初始决策图作为先验输入,学习图像丰富的细节信息;在损失函数中加入局部策略,包含结构相似度（structural similarity index measure,SSIM）和均方误差（mean squared error,MSE）,以确保更加准确地还原图像。结果 在Lytro等公开数据集上从主观和客观角度对本文模型进行评价,以验证融合算法设计的合理性。从主观评价来看,模型不仅可以较好地融合聚焦区域,有效避免融合图像中出现伪影,而且能够保留足够的细节信息,视觉效果自然清晰;从客观评价来看,通过将模型融合的图像与其他主流多聚焦图像融合算法的融合图像进行量化比较,在熵、Q_w、相关系数和视觉信息保真度上的平均精度均为最优,分别为7.457 4,0.917 7,0.978 8和0.890 8。结论 提出了一种用于多聚焦图像的融合算法,不仅能够对融合规则进行自学习、调整,并且融合图像效果可与现有方法媲美,有助于进一步理解基于深度学习的多聚焦图像融合机制。     

Boundary finding based multi-focus image fusion through multi-scale morphological focus-measure

Multi-focus image fusion aims to extract the focused regions from multiple partially focused images of the same scene and then combine them together to produce a completely focused image. Detecting the focused regions from multiple images is key for multi-focus image fusion. In this paper, we propose a novel boundary finding based multi-focus image fusion algorithm, in which the task of detecting the focused regions is treated as finding the boundaries between the focused and defocused regions from the source images. According to the found boundaries, the source images could be naturally separated into regions with the same focus conditions, i.e., each region is fully focused or defocused. Then, the focused regions can be found out by selecting the regions with greater focus-measures from each pair of regions. To improve the precision of boundary detection and focused region detection, we also present a multi-scale morphological focus-measure, effectiveness of which has been verified by using some quantitative evaluations. Different from the general multi-focus image fusion algorithms, our algorithm fuses the boundary regions and non-boundary regions of the source images respectively, which helps produce a fusion image with good visual quality. Moreover, the experimental results validate that the proposed algorithm outperforms some state-of-the-art image fusion algorithms in both qualitative and quantitative evaluations.     

MFF-GAN: An unsupervised generative adversarial network with adaptive and gradient joint constraints for multi-focus image fusion

Multi-focus image fusion is an enhancement method to generate full-clear images, which can address the depth-of-field limitation in imaging of optical lenses. Most existing methods generate the decision map to realize multi-focus image fusion, which usually lead to detail loss due to misclassification, especially near the boundary line of the focused and defocused regions. To overcome this challenge, this paper presents a new generative adversarial network with adaptive and gradient joint constraints to fuse multi-focus images. In our model, an adaptive decision block is introduced to determine whether source pixels are focused or not based on the difference of repeated blur. Under its guidance, a specifically designed content loss can dynamically guide the optimization trend, that is, force the generator to produce a fused result of the same distribution as the focused source images. To further enhance the texture details, we establish an adversarial game so that the gradient map of the fused result approximates the joint gradient map constructed based on the source images. Our model is unsupervised without requiring ground-truth fused images for training. In addition, we release a new dataset containing 120 high-quality multi-focus image pairs for benchmark evaluation. Experimental results demonstrate the superiority of our method over the state-of-the-art in terms of both subjective visual effect and quantitative metrics. Moreover, our method is about one order of magnitude faster compared with the state-of-the-art.     

基于EI优化权值的多聚焦图像融合方法

针对图像融合空间域方法中权值确定的问题,提出了一种基于图像融合客观评价准则中的边缘信息E(IEdge Information)的优化权值方法,通过寻找最优权值使得融合图像的EI最大,从而获得最好的融合效果。该方法采用适应度函数为EI的遗传算法实现,并且结合分块加权融合方法融合多聚焦图像,得到最优融合结果。实验结果表明,该方法有良好的融合效果且优于传统的融合方法。     

Novel multi-focus image fusion based on PCNN and random walks

The purpose of multi-focus image fusion is to acquire an image where all the objects are focused by fusing the source images which have different focus points. A novel multi-focus image fusion method is proposed in this paper, which is based on PCNN and random walks. PCNN is consistent with people’s visual perception. And the random walks model has been proven to have enormous potential to fuse image in recent years. The proposed method first employs PCNN to measure the sharpness of source images. Then, an original fusion map is constructed. Next, the method of random walks is employed to improve the accuracy of the fused regions detection. Finally, the fused image is generated according to the probability computed by random walks. The experiments demonstrate that our method outperforms many existing methods of multi-focus image fusion in visual perception and objective criteria. To assess the performance of our method in practical application, some examples are given at the end of paper.

     

利用局部极值点特性的多聚焦图像快速融合

目的：多聚焦图像融合技术一个关键问题是如何准确地判断待融合图像的清晰度。本文提出了基于归一化结构极值点数目的清晰度判断准则。方法：本文基于图像的局部极值点特性,定义了归一化结构极值点数目这个指标作为清晰度判断准则,同时还给出了利用该准则和融合决策矩阵快速估计技术的多聚焦图像快速融合方法。结果：利用本文提出的清晰度判断准则和融合方法,实验表明上述问题得到了很好的解决。结论：本文提出了一个新的图像清晰度判断准则,该准则判断准确率高,且对脉冲噪声有好的鲁棒性。通过与传统融合方法对两组实验图像融合结果的主客观比较表明,该方法的融合速度和效果比现有多聚焦图像融合方法有明显提高。     

基于遗传算法和形态学运算的多焦点图像融合

目前大多数图像融合算法将每个像素都独立对待，使像素之间关系割裂开来。本文提出了一种基于形态学算法和遗传算法的多焦点图像融合方法，此种方法有效地结合了像素级融合方法和特征级融合方法。其基本思想是先检测出原始图像中清晰聚焦的区域，再将这些区域提取出来，组成各部分都清晰聚焦的结果图像。实验结果证明，此方法优于Haar小波融合方法和形态学小波融合方法。特别是在原始图像没有完全配准的情况下，此种方法更为有效。     

Multi-focus image fusion using a morphology-based focus measure in a quad-tree structure

Finite depth-of-field poses a problem in light optical imaging systems since the objects present outside the range of depth-of-field appear blurry in the recorded image. Effective depth-of-field of a sensor can be enhanced considerably without compromising the quality of the image by combining multi-focus images of a scene. This paper presents a block-based algorithm for multi-focus image fusion. In general, finding a suitable block-size is a problem in block-based methods. A large block is more likely to contain portions from both focused and defocused regions. This may lead to selection of considerable amount of defocused regions. On the other hand, small blocks do not vary much in relative contrast and hence difficult to choose from. Moreover, small blocks are more affected by mis-registration problems. In this work, we present a block-based algorithm which do not use a fixed block-size and rather makes use of a quad-tree structure to obtain an optimal subdivision of blocks. Though the algorithm starts with blocks, it ultimately identifies sharply focused regions in input images. The algorithm is simple, computationally efficient and gives good results. A new focus-measure called energy of morphologic gradients is introduced and is used in the algorithm. It is comparable with other focus measures viz.energy of gradients, variance, Tenengrad, energy of Laplacian and sum modified Laplacian. The algorithm is robust since it works with any of the above focus measures. It is also robust against pixel mis-registration. Performance of the algorithm has been evaluated by using two different quantitative measures.     

Edge and Depth from Focus

This paper proposes a novel method to obtain the reliable edge and depth information by integrating a set of multi-focus images, i.e., a sequence of images taken by systematically varying a camera parameter focus. In previous work on depth measurement using focusing or defocusing, the accuracy depends upon the size and location of local windows where the amount of blur is measured. In contrast, no windowing is needed in our method; the blur is evaluated from the intensity change along corresponding pixels in the multi-focus images. Such a blur analysis enables us not only to detect the edge points without using spatial differentiation but also to estimate the depth with high accuracy. In addition, the analysis result is stable because the proposed method involves integral computations such as summation and least-square model fitting. This paper first discusses the fundamental properties of multi-focus images based on a step edge model. Then, two algorithms are presented: edge detection using an accumulated defocus image which represents the spatial distribution of blur, and depth estimation using a spatio-focal image which represents the intensity distribution along focus axis. The experimental results demonstrate that the highly precise measurement has been achieved: 0.5 pixel position fluctuation in edge detection and 0.2% error at 2.4 m in depth estimation.     

采用shearlet变换的多聚焦图像融合

相对传统多尺度分析工具,shearlet变换更适于提取图像细节信息。采用shearlet变换进行图像融合,对源图像进行shearlet域分解,对低频子带采用SML算子作为融合依据,高频子带采取区域能量与单个像素相结合的方式选择系数,对融合后的系数进行逆shearlet变换得到融合图像。仿真实验表明,算法在视觉效果和量化结果上均有提高。     

Surface area-based focus criterion for multi-focus image fusion

Nowadays image processing and machine vision fields have become important research topics due to numerous applications in almost every field of science. Performance in these fields is critically dependent to the quality of input images. In most of the imaging devices, optical lenses are used to capture images from a particular scene. But due to the limited depth of field of optical lenses, objects in different distances from focal point will be captured with different sharpness and details. Thus, important details of the scene might be lost in some regions. Multi-focus image fusion is an effective technique to cope with this problem. The main challenge in multi-focus fusion is the selection of an appropriate focus measure. In this paper, we propose a novel focus measure based on the surface area of regions surrounded by intersection points of input source images. The potential of this measure to distinguish focused regions from the blurred ones is proved. In our fusion algorithm, intersection points of input images are calculated and then input images are segmented using these intersection points. After that, the surface area of each segment is considered as a measure to determine focused regions. Using this measure we obtain an initial selection map of fusion which is then refined by morphological modifications. To demonstrate the performance of the proposed method, we compare its results with several competing methods. The results show the effectiveness of our proposed method.     

基于引导滤波和差分图像的多聚焦图像融合方法

针对传统的多聚焦图像的空间域融合容易出现边缘模糊的问题,提出了一种基于引导滤波（GF）和差分图像的多聚焦图像融合方法。首先,将源图像进行不同水平的GF,并对滤波后图像进行差分,从而获得聚焦特征图像;随后,利用聚焦特征图像的梯度能量（EOG）信息获得初始决策图,对初始决策图进行空间一致性检查以及形态学操作以消除因EOG相近而造成的噪点;然后,对初始决策图进行GF以得到优化后决策图,从而避免融合后的图像存在边缘骤变的问题;最后,基于优化后决策图对源图像进行加权融合,以得到融合图像。选取3组经典的多聚焦图像作为实验图像,将所提方法与其他9种多聚焦图像融合方法得到的结果进行比较。主观视觉效果显示,所提方法能更好地将多聚焦图像的细节信息保存下来,另外,经该方法处理后的图像的4项客观评价指标均显著优于对比方法。结果表明,所提方法能够获得高质量的融合图像,较好地保留原始图像信息,有效解决传统多聚焦图像融合出现的边缘模糊问题。     

基于快速清晰度估计的多聚焦图像融合

为了准确地估计源图像的清晰区域,提高多聚焦图像融合的效率,本文提出了一种新的基于清晰度估计的图像融合方法。首先,利用基于离散小波的清晰度估计方法获取源图像的聚焦区域,然后使用均值滤波和空洞填充进一步优化该聚焦区域,最后结合清晰度估计和相似性特性,将不同聚焦区域合并生成融合图像。该方法获得的融合图像在客观评价和主观质量上都优于以往基于清晰度的图像融合方法。     

基于注意力机制的多尺度融合航拍影像语义分割

航拍影像同一场景不同对象尺度差异较大,采用单一尺度的分割往往无法达到最 佳的分类效果。为解决这一问题,提出一种基于注意力机制的多尺度融合模型。首先,利用不 同采样率的扩张卷积提取航拍影像的多个尺度特征;然后,在多尺度融合阶段引入注意力机制, 使模型能够自动聚焦于合适的尺度,并为所有尺度及每个位置像素分别赋予权重;最后,将加 权融合后的特征图上采样到原图大小,对航拍影像的每个像素进行语义标注。实验结果表明, 与传统的 FCN、DeepLab 语义分割模型及其他航拍影像分割模型相比,基于注意力机制的多尺 度融合模型不仅具有更高的分割精度,而且可以通过对各尺度特征对应权重图的可视化,分析 不同尺度及位置像素的重要性。     

基于二次成像与清晰度差异的多聚焦图像融合

本文提出了一种基于清晰度差异的不同聚焦点图像的融合方法。该方法首先选择了一种基于梯度向量模方和的清晰度定义,然后根据几何光学系统的成像模型,以及点扩散函数的作用效果提出了模拟光学系统的二次成像模型。然后根据二次成像前后各图像清晰度的差异情况,对各幅图像中的目标进行判断,并选择其中的清晰部分生成融合图像。实验结果表明,该方法可以提取出多聚焦图像中的清晰目标,生成的融合图像效果优于Laplacian塔型方法和小波变换方法。     

基于冗余小波变换与引导滤波的多聚焦图像融合

针对传统多尺度变换在多聚焦图像融合中存在的边缘晕圈问题,提出了一种基于冗余小波变换与引导滤波的多聚焦图像融合算法。首先,利用冗余小波变换对图像进行多尺度分解,将源图像分解为一个相似平面和一系列小波平面,该多尺度分解能够有效地提取源图像中的细节信息;然后,对相似平面和小波平面分别采用引导滤波的加权融合规则来构造加权映射,从而得到相似平面和小波平面的加权融合系数;最后,进行冗余小波逆变换,即可得到融合结果图。实验结果表明,与传统融合算法相比,所提算法能够更好地体现图像边缘的细节特征,取得了较好的融合效果。