改进残差密集生成对抗网络的红外与可见光图像融合

基于深度学习的红外与可见光图像融合算法通常无法感知源图像显著性区域,导致融合结果没有突出红外与可见光图像各自的典型特征,无法达到理想的融合效果.针对上述问题,设计一种适用于红外与可见光图像融合任务的改进残差密集生成对抗网络结构.首先,使用改进残差密集模块作为基础网络分别构建生成器与判别器,并引入基于注意力机制的挤压激励网络来捕获通道维度下的显著特征,充分保留红外图像的热辐射信息和可见光图像的纹理细节信息;其次,使用相对平均判别器,分别衡量融合图像与红外图像、可见光图像之间的相对差异,并根据差异指导生成器保留缺少的源图像信息;最后,在TNO等多个图像融合数据集上进行实验,结果表明所提方法能够生成目标清晰、细节丰富的融合图像,相比基于残差网络的融合方法,边缘强度和平均梯度分别提升了64.56%和64.94%.     

Boosting attention fusion generative adversarial network for image denoising

Neural Computing and Applications - Boosting has received considerable attention to improve the overall performance of model in multiple tasks by cascading many steerable sub-modules. In this...     

Unified gradient- and intensity-discriminator generative adversarial network for image fusion

This study proposes a unified gradient- and intensity-discriminator generative adversarial network for various image fusion tasks, including infrared and visible image fusion, medical image fusion, multi-focus image fusion, and multi-exposure image fusion. On the one hand, we unify all fusion tasks into discriminating a fused image’s gradient and intensity distributions based on a generative adversarial network. The generator adopts a dual-encoder–single-decoder framework to extract source image features by using different encoder paths. A dual-discriminator is employed to distinguish the gradient and intensity, ensuring that the generated image contains the desired geometric structure and conspicuous information. The dual adversarial game can tackle the generative adversarial network’s mode collapse problem. On the other hand, we define a loss function based on the gradient and intensity that can be adapted to various fusion tasks by using varying relevant parameters with the source images. Qualitative and quantitative experiments on publicly available datasets demonstrate our method’s superiority over state-of-the-art methods.     

A generative adversarial network for image denoising

Multimedia Tools and Applications - Recent studies have shown that the performance of image denoising methods can be improved significantly by using deep convolutional neural networks(CNN). The...     

UIFGAN: An unsupervised continual-learning generative adversarial network for unified image fusion

In this paper, we propose a novel unsupervised continual-learning generative adversarial network for unified image fusion, termed as UIFGAN. In our model, for multiple image fusion tasks, a generative adversarial network for training a single model with memory in a continual-learning manner is proposed, rather than training an individual model for each fusion task or jointly training multiple tasks. We use elastic weight consolidation to avoid forgetting what has been learned from previous tasks when training multiple tasks sequentially. In each task, the generation of the fused image comes from the adversarial learning between a generator and a discriminator. Meanwhile, a max-gradient loss function is adopted for forcing the fused image to obtain richer texture details of the corresponding regions in two source images, which applies to most typical image fusion tasks. Extensive experiments on multi-exposure, multi-modal and multi-focus image fusion tasks demonstrate the advantages of our method over the state-of-the-art approaches.     

生成对抗残差网络的医学图像融合算法

针对传统医学图像融合中需要依靠先验知识手动设置融合规则和参数,导致融合效果存在不确定性、细节表现力不足的问题,提出了一种基于改进生成对抗网络（GAN）的脑部计算机断层扫描（CT）/磁共振（MR）图像融合算法。首先,对生成器和判别器两个部分的网络结构进行改进,在生成器网络的设计中采用残差块和快捷连接以加深网络结构,更好地捕获深层次的图像信息;然后,去掉常规网络中的下采样层,以减少图像传输过程中的信息损失,并将批量归一化改为层归一化,以更好地保留源图像信息,增加判别器网络的深度以提高网络性能;最后,连接CT图像和MR图像,将其输入到生成器网络中得到融合图像,通过损失函数不断优化网络参数,训练出最适合医学图像融合的模型来生成高质量的图像。实验结果表明,与当前表现优良的基于离散小波变换（DWT）算法、基于非下采样剪切波变换（NSCT）算法、基于稀疏表示（SR）算法和基于图像分类块稀疏表示（PSR）算法对比,所提算法在互信息（MI）、信息熵（IE）、结构相似性（SSIM）上均表现良好,最终的融合图像纹理和细节丰富,同时避免了人为因素对融合效果稳定性的影响。     

门控融合对抗网络的水下图像增强

针对水下成像中图像存在的色彩失真、对比度低和雾化严重等问题,提出一种门控融合对抗网络的水下图像增强方法.主要特点是使用生成网络将图像特征信息逐像素恢复,并通过门控融合形成清晰图像.首先,利用多个并行子网络对同幅图像的不同空间特征信息进行学习,提升网络对图像特征学习的多样性.然后,通过门控融合,将不同子网络学习到的图像特...     

红外与可见光图像渐进融合深度网络

目的 红外与可见光图像融合的目标是获得具有完整场景表达能力的高质量融合图像。由于深度特征具有良好的泛化性、鲁棒性和发展潜力,很多基于深度学习的融合方法被提出,在深度特征空间进行图像融合,并取得了良好的效果。此外,受传统基于多尺度分解的融合方法的启发,不同尺度的特征有利于保留源图像的更多信息。基于此,提出了一种新颖的渐进式红外与可见光图像融合框架(progressive fusion, ProFuse)。方法 该框架以U-Net为骨干提取多尺度特征,然后逐渐融合多尺度特征,既对包含全局信息的高层特征和包含更多细节的低层特征进行融合,也在原始尺寸特征(保持更多细节)和其他更小尺寸特征(保持语义信息)上进行融合,最终逐层重建融合图像。结果 实验在TNO(Toegepast Natuurwetenschappelijk Onderzoek)和INO(Institut National D’optique)数据集上与其他6种方法进行比较,在选择的6项客观指标上,本文方法在互信息(mutual Information, MI)上相比FusionGAN(generative adversarial ...     

RXDNFuse: A aggregated residual dense network for infrared and visible image fusion

This study proposes a novel unsupervised network for IR/VIS fusion task, termed as RXDNFuse, which is based on the aggregated residual dense network. In contrast to conventional fusion networks, RXDNFuse is designed as an end-to-end model that combines the structural advantages of ResNeXt and DenseNet. Hence, it overcomes the limitations of the manual and complicated design of activity-level measurement and fusion rules. Our method establishes the image fusion problem into the structure and intensity proportional maintenance problem of the IR/VIS images. Using comprehensive feature extraction and combination, RXDNFuse automatically estimates the information preservation degrees of corresponding source images, and extracts hierarchical features to achieve effective fusion. Moreover, we design two loss function strategies to optimize the similarity constraint and the network parameter training, thus further improving the quality of detailed information. We also generalize RXDNFuse to fuse images with different resolutions and RGB scale images. Extensive qualitative and quantitative evaluations reveal that our results can effectively preserve the abundant textural details and the highlighted thermal radiation information. In particular, our results form a comprehensive representation of scene information, which is more in line with the human visual perception system.     

A physics based generative adversarial network for single image defogging

In the field of single image defogging, there are two main methods. One is the image restoration method based on the atmospheric scattering theory which can recover the image texture details well. The other is the image enhancement method based on Retinex theory which can improve the image contrast well. In practice, however, the former can easily lead to low contrast images; the latter is prone to losing texture details. Therefore, how to effectively combine the advantages of both to remove fog is a key issue in the field. In this paper, we have developed a physics based generative adversarial network (PBGAN) to exploit the advantages between those two methods in parallel. To our knowledge, it is the first learning defogging framework that incorporates these two methods and to enable them to work together and complement each other. Our method has two generative adversarial modules, the Contrast Enhancement (CE) module and the Texture Restoration (TR) module. To improve contrast in the CE module, we introduced a novel inversion-adversarial loss and a novel inversion-cycle consistency loss for training the generator. To improve the texture in the TR module, we introduced two convolutional neural networks to learn the atmospheric light coefficient and the transmission map, respectively. Extensive experiments on both synthetic and real-world datasets demonstrate that the proposed approach performs better than several state-of-the-art methods quantitatively and qualitatively.     

类别信息生成式对抗网络的单图超分辨重建

目的 基于生成式对抗网络的超分辨模型（SRGAN）以感知损失函数作为优化目标,有效解决了传统基于均方误差（MSE）的损失函数导致重建图像模糊的问题。但是SRGAN的感知损失函数中并未添加明确指示模型生成对应特征的标志性信息,使得其无法精准地将数据的具体维度与语义特征对应起来,受此局限性影响,模型对于生成图像的特征信息表示不足,导致重建结果特征不明显,给后续识别处理过程带来困难。针对上述问题,在SRGAN方法的基础上,提出一种类别信息生成式对抗网络的超分辨模型（class-info SRGAN）。方法 对SRGAN模型增设类别分类器,并将类别损失项添加至生成网络损失中,再利用反向传播训练更新网络参数权重,以达到为模型提供特征类别信息的目的,最终生成具有可识别特征的重建图像。创新及优势在于将特征类别信息引入损失函数,改进了超分辨模型的优化目标,使得重建结果的特征表示更加突出。结果 经CelebA数据集测试表明：添加性别分类器的class-info SRGAN的生成图像性别特征识别率整体偏高（58%97%）;添加眼镜分类器的class-info SRGAN的生成图像眼镜框架更加清晰。此外,模型在Fashion-mnist与Cifar-10数据集上的结果同样表明其相较于SRGAN的重建质量更佳。结论 实验结果验证了本方法在超分辨重建任务中的优势和有效性,同时结果显示：虽然class-info SRGAN更适用于具有简单、具体属性特征的图像,但总体而言仍是一种效果显著的超分辨模型。     

Triple discriminator generative adversarial network for zero-shot image classification

One key challenge in zero-shot classification(ZSC)is the exploration of knowledge hidden in unseen classes.Generative methods such as generative adversarial net...     

MFF-GAN: An unsupervised generative adversarial network with adaptive and gradient joint constraints for multi-focus image fusion

Multi-focus image fusion is an enhancement method to generate full-clear images, which can address the depth-of-field limitation in imaging of optical lenses. Most existing methods generate the decision map to realize multi-focus image fusion, which usually lead to detail loss due to misclassification, especially near the boundary line of the focused and defocused regions. To overcome this challenge, this paper presents a new generative adversarial network with adaptive and gradient joint constraints to fuse multi-focus images. In our model, an adaptive decision block is introduced to determine whether source pixels are focused or not based on the difference of repeated blur. Under its guidance, a specifically designed content loss can dynamically guide the optimization trend, that is, force the generator to produce a fused result of the same distribution as the focused source images. To further enhance the texture details, we establish an adversarial game so that the gradient map of the fused result approximates the joint gradient map constructed based on the source images. Our model is unsupervised without requiring ground-truth fused images for training. In addition, we release a new dataset containing 120 high-quality multi-focus image pairs for benchmark evaluation. Experimental results demonstrate the superiority of our method over the state-of-the-art in terms of both subjective visual effect and quantitative metrics. Moreover, our method is about one order of magnitude faster compared with the state-of-the-art.     

Multi-focus images fusion via residual generative adversarial network

Multimedia Tools and Applications - Recently, most existing learning-based fusion methods are not fully end-to-end, which still predict the decision map and recover the fused image by the refined...     

Multiscale aggregation and illumination-aware attention network for infrared and visible image fusion

Image fusion plays a significant role in computer vision since numerous applications benefit from the fusion results. The existing image fusion methods are incapable of perceiving the most discriminative regions under varying illumination circumstances and thus fail to emphasize the salient targets and ignore the abundant texture details of the infrared and visible images. To address this problem, a multiscale aggregation and illumination-aware attention network (MAIANet) is proposed for infrared and visible image fusion. Specifically, the MAIANet consists of four modules, namely multiscale feature extraction module, lightweight channel attention module, image reconstruction module, and illumination-aware module. The multiscale feature extraction module attempts to extract multiscale features in the images. The role of the lightweight channel attention module is to assign different weights to each channel so as to focus on the essential regions in the infrared and visible images. An illumination-aware module is employed to assess the probability distribution regarding the illumination factor. Meanwhile, an illumination perception loss is formulated by the illumination probabilities to enable the proposed MAIANet to better adjust to the changes in illumination. Experimental results on three datasets, that is, MSRS, TNO, and RoadSence, verify the effectiveness of the MAIANet in both qualitative and quantitative evaluations.     

基于生成式对抗网络的图像自动标注

针对基于深度学习的图像标注模型输出层神经元数目与标注词汇量成正比，导致模型结构因词汇量的变化而改变的问题，提出了结合生成式对抗网络（GAN）和Word2vec的新标注模型。首先，通过Word2vec将标注词汇映射为固定的多维词向量；其次，利用GAN构建神经网络模型--GAN-W模型，使输出层神经元数目与多维词向量维数相等，与词汇量不再相关；最后，通过对模型多次输出结果的排序来确定最终标注。GAN-W模型分别在Corel 5K和IAPRTC-12图像标注数据集上进行实验，在Corel 5K数据集上，GAN-W模型准确率、召回率和F1值比卷积神经网络回归（CNN-R）方法分别提高5、14和9个百分点；在IAPRTC-12数据集上，GAN-W模型准确率、召回率和F1值比两场K最邻近（2PKNN）模型分别提高2、6和3个百分点。实验结果表明，GAN-W模型可以解决输出神经元数目随词汇量改变的问题，同时每幅图像标注的标签数目自适应，使得该模型标注结果更加符合实际标注情形。     

PIAFusion: A progressive infrared and visible image fusion network based on illumination aware

Infrared and visible image fusion aims to synthesize a single fused image containing salient targets and abundant texture details even under extreme illumination conditions. However, existing image fusion algorithms fail to take the illumination factor into account in the modeling process. In this paper, we propose a progressive image fusion network based on illumination-aware, termed as PIAFusion, which adaptively maintains the intensity distribution of salient targets and preserves texture information in the background. Specifically, we design an illumination-aware sub-network to estimate the illumination distribution and calculate the illumination probability. Moreover, we utilize the illumination probability to construct an illumination-aware loss to guide the training of the fusion network. The cross-modality differential aware fusion module and halfway fusion strategy completely integrate common and complementary information under the constraint of illumination-aware loss. In addition, a new benchmark dataset for infrared and visible image fusion, i.e., Multi-Spectral Road Scenarios (available at https://github.com/Linfeng-Tang/MSRS), is released to support network training and comprehensive evaluation. Extensive experiments demonstrate the superiority of our method over state-of-the-art alternatives in terms of target maintenance and texture preservation. Particularly, our progressive fusion framework could round-the-clock integrate meaningful information from source images according to illumination conditions. Furthermore, the application to semantic segmentation demonstrates the potential of our PIAFusion for high-level vision tasks. Our codes will be available at https://github.com/Linfeng-Tang/PIAFusion.     

EnGAN：医学图像分割中的增强生成对抗网络

原始采集的医学图像普遍存在对比度不足、细节模糊以及噪声干扰等质量问题,使得现有医学图像分割技术的精度很难达到新的突破。针对医学图像数据增强技术进行研究,在不明显改变图像外观的前提下,通过添加特定的像素补偿和进行细微的图像调整来改善原始图像质量问题,从而提高图像分割准确率。首先,设计引入了一个新的优化器模块,以产生一个连续分布的空间作为迁移的目标域,该优化器模块接受数据集的标签作为输入,并将离散的标签数据映射到连续分布的医学图像中;其次,提出了一个基于对抗生成网络的EnGAN模型,并将优化器模块产生的迁移目标域用来指导对抗网络的目标生成,从而将改善的医学图像质量知识植入模型中实现图像增强。基于COVID-19数据集,实验中使用U-Net、U-Net+ResNet34、U-Net+Attn Res U-Net等卷积神经网络作为骨干网络,Dice系数和交并比分别达到了73.5%和69.3%、75.1%和70.5%,以及75.2%和70.3%。实验的结果表明,提出的医学图像质量增强技术在最大限度保留原始特征的条件下,有效地提高了分割的准确率,为后续的医学图像处理研究提供了一个更为稳健和高效的解决方案。     

SDGAN: A novel spatial deformable generative adversarial network for low-dose CT image reconstruction

X-ray computed tomography (CT) is one of the most commonly used medical imaging technologies for the evaluation of many diseases. Full-dose imaging for CT ensures the image quality but usually raises concerns about the potential health risks of radiation exposure, especially for cancer patients. The conflict between reducing the radiation exposure and remaining diagnostic performance can be addressed effectively via reconstructing the low-dose CT (L-CT) to the high-quality full-dose CT (F-CT) ones. In this paper, we propose a Spatial Deformable Generative Adversarial Network (SDGAN) to achieve efficient L-CT reconstruction and analysis. SDGAN consists of three modules: the conformer-based generator, the dual-scale discriminator and the spatial deformable fusion module (SDFM). A sequence of consecutive L-CT slices is first fed into the conformer-based generator with the dual-scale discriminator to generate F-CT images. These estimated F-CT images are then fed into SDFM, which fully explores the inter- and intra-slice spatial information, to synthesize the final F-CT images with high quality. Experimental results show that the proposed SDGAN achieves the state-of-the-art performances on commonly used metrics and satisfies the reconstruction needs for clinical standards.     

一种新的红外与可见光图像融合算法

提出了一种新的红外与可见光图像融合算法,首先应用非抽样Contourlet变换(NSCT)对图像进行多尺度、多方向变换,对变换的低频子带采用改进的能量加权法融合,带通子带融合采用最大系数与区域方差加权相结合方法,然后对融合的2个子带系数进行NSCT反变换,得到融合图像。对不同算法的融合实验结果进行比较,通过主观和客观评价,该算法融合效果较好。