Unified gradient- and intensity-discriminator generative adversarial network for image fusion

This study proposes a unified gradient- and intensity-discriminator generative adversarial network for various image fusion tasks, including infrared and visible image fusion, medical image fusion, multi-focus image fusion, and multi-exposure image fusion. On the one hand, we unify all fusion tasks into discriminating a fused image’s gradient and intensity distributions based on a generative adversarial network. The generator adopts a dual-encoder–single-decoder framework to extract source image features by using different encoder paths. A dual-discriminator is employed to distinguish the gradient and intensity, ensuring that the generated image contains the desired geometric structure and conspicuous information. The dual adversarial game can tackle the generative adversarial network’s mode collapse problem. On the other hand, we define a loss function based on the gradient and intensity that can be adapted to various fusion tasks by using varying relevant parameters with the source images. Qualitative and quantitative experiments on publicly available datasets demonstrate our method’s superiority over state-of-the-art methods.     

MFF-GAN: An unsupervised generative adversarial network with adaptive and gradient joint constraints for multi-focus image fusion

Multi-focus image fusion is an enhancement method to generate full-clear images, which can address the depth-of-field limitation in imaging of optical lenses. Most existing methods generate the decision map to realize multi-focus image fusion, which usually lead to detail loss due to misclassification, especially near the boundary line of the focused and defocused regions. To overcome this challenge, this paper presents a new generative adversarial network with adaptive and gradient joint constraints to fuse multi-focus images. In our model, an adaptive decision block is introduced to determine whether source pixels are focused or not based on the difference of repeated blur. Under its guidance, a specifically designed content loss can dynamically guide the optimization trend, that is, force the generator to produce a fused result of the same distribution as the focused source images. To further enhance the texture details, we establish an adversarial game so that the gradient map of the fused result approximates the joint gradient map constructed based on the source images. Our model is unsupervised without requiring ground-truth fused images for training. In addition, we release a new dataset containing 120 high-quality multi-focus image pairs for benchmark evaluation. Experimental results demonstrate the superiority of our method over the state-of-the-art in terms of both subjective visual effect and quantitative metrics. Moreover, our method is about one order of magnitude faster compared with the state-of-the-art.     

UIFGAN: An unsupervised continual-learning generative adversarial network for unified image fusion

In this paper, we propose a novel unsupervised continual-learning generative adversarial network for unified image fusion, termed as UIFGAN. In our model, for multiple image fusion tasks, a generative adversarial network for training a single model with memory in a continual-learning manner is proposed, rather than training an individual model for each fusion task or jointly training multiple tasks. We use elastic weight consolidation to avoid forgetting what has been learned from previous tasks when training multiple tasks sequentially. In each task, the generation of the fused image comes from the adversarial learning between a generator and a discriminator. Meanwhile, a max-gradient loss function is adopted for forcing the fused image to obtain richer texture details of the corresponding regions in two source images, which applies to most typical image fusion tasks. Extensive experiments on multi-exposure, multi-modal and multi-focus image fusion tasks demonstrate the advantages of our method over the state-of-the-art approaches.     

基于多特征融合的多尺度生成对抗网络图像修复算法

针对多尺度生成式对抗网络图像修复算法（MGANII）在修复图像过程中训练不稳定、修复图像的结构一致性差以及细节和纹理不足等问题,提出了一种基于多特征融合的多尺度生成对抗网络的图像修复算法。首先,针对结构一致性差以及细节和纹理不足的问题,在传统的生成器中引入多特征融合模块（MFFM）,并且引入了一个基于感知的特征重构损失函数来提高扩张卷积网络的特征提取能力,从而改善修复图像的细节性和纹理特征;然后,在局部判别器中引入了一个基于感知的特征匹配损失函数来提升判别器的鉴别能力,从而增强了修复图像的结构一致性;最后,在对抗损失函数中引入风险惩罚项来满足利普希茨连续条件,使得网络在训练过程中能快速稳定地收敛。在CelebA数据集上,所提的多特征融合的图像修复算法与MANGII相比能快速收敛,同时所提算法所修复图像的峰值信噪比（PSNR）、结构相似性（SSIM）比基线算法所修复图像分别提高了0.45%～8.67%和0.88%～8.06%,而Frechet Inception距离得分（FID）比基线算法所修复图像降低了36.01%～46.97%。实验结果表明,所提算法的修复性能优于基线算法。     

生成对抗残差网络的医学图像融合算法

针对传统医学图像融合中需要依靠先验知识手动设置融合规则和参数,导致融合效果存在不确定性、细节表现力不足的问题,提出了一种基于改进生成对抗网络（GAN）的脑部计算机断层扫描（CT）/磁共振（MR）图像融合算法。首先,对生成器和判别器两个部分的网络结构进行改进,在生成器网络的设计中采用残差块和快捷连接以加深网络结构,更好地捕获深层次的图像信息;然后,去掉常规网络中的下采样层,以减少图像传输过程中的信息损失,并将批量归一化改为层归一化,以更好地保留源图像信息,增加判别器网络的深度以提高网络性能;最后,连接CT图像和MR图像,将其输入到生成器网络中得到融合图像,通过损失函数不断优化网络参数,训练出最适合医学图像融合的模型来生成高质量的图像。实验结果表明,与当前表现优良的基于离散小波变换（DWT）算法、基于非下采样剪切波变换（NSCT）算法、基于稀疏表示（SR）算法和基于图像分类块稀疏表示（PSR）算法对比,所提算法在互信息（MI）、信息熵（IE）、结构相似性（SSIM）上均表现良好,最终的融合图像纹理和细节丰富,同时避免了人为因素对融合效果稳定性的影响。     

基于条件Wassertein生成对抗网络的图像生成

生成对抗网络(GAN)能够自动生成目标图像,对相似地块的建筑物排布生成具有重要意义.而目前训练模型的过程中存在生成图像精度不高、模式崩溃、模型训练效率太低的问题.针对这些问题,提出了一种面向图像生成的条件Wassertein生成对抗网络(C-WGAN)模型.首先,该模型需要识别真实样本和目标样本之间特征对应关系,然后,...     

门控融合对抗网络的水下图像增强

针对水下成像中图像存在的色彩失真、对比度低和雾化严重等问题,提出一种门控融合对抗网络的水下图像增强方法.主要特点是使用生成网络将图像特征信息逐像素恢复,并通过门控融合形成清晰图像.首先,利用多个并行子网络对同幅图像的不同空间特征信息进行学习,提升网络对图像特征学习的多样性.然后,通过门控融合,将不同子网络学习到的图像特...     

基于多姿态特征融合生成对抗网络的人脸校正方法

针对人脸校正中单幅图像难以解决大姿态侧脸的问题，提出一种基于多姿态特征融合生成对抗网络（MFFGAN）的人脸校正方法，利用多幅不同姿态侧脸之间的相关信息来进行人脸校正，并采用对抗机制对网络参数进行调整。该方法设计了一种新的网络，包括由多姿态特征提取、多姿态特征融合、正脸合成三个模块组成的生成器，以及用于对抗训练的判别器。多姿态特征提取模块利用多个卷积层提取侧脸图像的多姿态特征；多姿态特征融合模块将多姿态特征融合成包含多姿态侧脸信息的融合特征；而正脸合成模块在进行姿态校正的过程中加入融合特征，通过探索多姿态侧脸图像之间的特征依赖关系来获取相关信息与全局结构，可以有效提高校正结果。实验结果表明，与现有基于深度学习的人脸校正方法相比，所提方法恢复出的正脸图像不仅轮廓清晰，而且从两幅侧脸中恢复出的正脸图像的识别率平均提高了1.9个百分点，并且输入侧脸图像越多，恢复出的正脸图像的识别率越高，表明所提方法可以有效融合多姿态特征来恢复出轮廓清晰的正脸图像。     

基于多姿态特征融合生成对抗网络的人脸校正方法

针对人脸校正中单幅图像难以解决大姿态侧脸的问题,提出一种基于多姿态特征融合生成对抗网络（MFFGAN）的人脸校正方法,利用多幅不同姿态侧脸之间的相关信息来进行人脸校正,并采用对抗机制对网络参数进行调整。该方法设计了一种新的网络,包括由多姿态特征提取、多姿态特征融合、正脸合成三个模块组成的生成器,以及用于对抗训练的判别器。多姿态特征提取模块利用多个卷积层提取侧脸图像的多姿态特征;多姿态特征融合模块将多姿态特征融合成包含多姿态侧脸信息的融合特征;而正脸合成模块在进行姿态校正的过程中加入融合特征,通过探索多姿态侧脸图像之间的特征依赖关系来获取相关信息与全局结构,可以有效提高校正结果。实验结果表明,与现有基于深度学习的人脸校正方法相比,所提方法恢复出的正脸图像不仅轮廓清晰,而且从两幅侧脸中恢复出的正脸图像的识别率平均提高了1.9个百分点,并且输入侧脸图像越多,恢复出的正脸图像的识别率越高,表明所提方法可以有效融合多姿态特征来恢复出轮廓清晰的正脸图像。     

基于双通道循环一致性GAN的跨视角步态识别研究

步态识别系统在生物识别领域显示出巨大的潜力,然而步态识别的准确性很容易受到视角的影响。为解决这一问题,提出的方法基于循环生成对抗网络(cycle generative adversarial network, Cycle-GAN)的网络结构,结合新的特征提取模块以及多重损失函数,提出了一种用于跨视角步态识别的网络模型双通道循环一致性生成对抗网络(two-channel cycle consistency generative adversarial network, TCC-GAN)。该网络首先将步态能量图像从任意视角转换为目标视角图像,然后进行比对从而实现身份识别。TCC-GAN分别引入了重建损失、视角分类和身份保持损失来指导生成器生成具有目标视角的步态图像并同时保留身份信息。为了避免可能存在的模式崩塌问题,并保证各个输入和输出以有意义的方式进行映射,模型中还利用了循环一致性损失。数据集CASIA-B和OU-MVLP上的实验结果表明:所提TCC-GAN模型的跨视角识别率高于目前大多数其他基于GAN的跨视角步态识别模型。     

基于多尺度残差生成对抗网络的单图像盲去运动模糊方法

图像盲去运动模糊一直是计算机视觉领域的一个经典问题,它的目的是在模糊核未知的情况下恢复清晰图像。考虑到更大的感受野以及多尺度信息对恢复清晰图像中的全局信息以及局部细节信息具有重要作用,因此提出的方法对DeblurGAN方法进行改进,提出一种基于条件生成对抗网络的GR-DeblurGAN(granular resi-dual DeblurGAN)的单图像盲去运动模糊方法,采用细粒度残差模块(granular residual block)作为骨干网络,以此在不增加参数量的情况下,扩大感受野,获得多尺度信息。最后在两个广泛使用的数据集:GoPro数据集以及Kohler数据集上进行算法性能评估,并与代表性算法进行对比。从实验结果可以看出,提出的方法改进效果明显,并且在计算开销上面优于其他算法。     

多模式生成对抗网络

生成对抗网络已经成为深度学习领域最热门的研究方向之一,其最大的优势在于能够以无监督的方式来拟合一个未知的分布。目前,生成对抗网络在图像生成领域大放异彩,其能够产生一些高质量的图像,但也暴露了一些弊端。在生成图像的过程中,经常会出现模式坍塌问题,从而导致生成的样本过于单一。为了解决这个问题,对生成对抗网络的模型结构和损失函数加以改进,使判别器能够从多个角度来度量生成数据的分布和真实数据的分布之间的差异,从而改善了生成样本的多样性。通过在多个数据集上进行实验,结果显示,提出的模型在很大程度上缓解了模式坍塌问题。     

生成对抗网络下的低剂量CT图像增强

为去除低剂量计算机断层扫描（LDCT）图像中的噪声,增强去噪后图像的显示效果,提出一种生成对抗网络（GAN）下的LDCT图像增强算法。首先,将GAN与感知损失、结构损失相结合对LDCT图像进行去噪;然后,对去噪后的图像分别进行动态灰度增强和边缘轮廓增强;最后,利用非下采样轮廓波变换（NSCT）将增强后的图像在频域上分解为具有多方向性的系数子图,并将配对的高低频子图使用卷积神经网络（CNN）进行自适应融合,以重构得到增强后的计算机断层扫描（CT）图像。使用AAPM比赛公开的真实临床数据作为实验数据集,进行图像去噪、增强、融合实验,所提方法在峰值信噪比（PSNR）、结构相似度（SSIM）和均方根误差（RMSE）上的结果分别为33.015 5 dB、0.918 5和5.99。实验结果表明,所提算法在去除噪声的同时能保留CT图像的细节信息,提高图像的亮度和对比度,有助于医生更加准确地分析病情。     

Multimodal deep generative adversarial models for scalable doubly semi-supervised learning

The comprehensive utilization of incomplete multi-modality data is a difficult problem with strong practical value. Most of the previous multimodal learning algorithms require massive training data with complete modalities and annotated labels, which greatly limits their practicality. Although some existing algorithms can be used to complete the data imputation task, they still have two disadvantages: (1) they cannot control the semantics of the imputed modalities accurately; and (2) they need to establish multiple independent converters between any two modalities when extended to multimodal cases. To overcome these limitations, we propose a novel doubly semi-supervised multimodal learning (DSML) framework. Specifically, DSML uses a modality-shared latent space and multiple modality-specific generators to associate multiple modalities together. Here we divided the shared latent space into two independent parts, the semantic labels and the semantic-free styles, which allows us to easily control the semantics of generated samples. In addition, each modality has its own separate encoder and classifier to infer the corresponding semantic and semantic-free latent variables. The above DSML framework can be adversarially trained by using our specially designed softmax-based discriminators. Large amounts of experimental results show that the DSML obtains better performance than the baselines on three tasks, including semi-supervised classification, missing modality imputation and cross-modality retrieval.     

基于生成对抗网络的红外图像数据增强

深度学习在视觉任务中的良好表现很大程度上依赖于海量的数据和计算力的提升,但是在很多实际项目中通常难以提供足够的数据来完成任务。针对某些情况下红外图像少且难以获得的问题,提出一种基于彩色图像生成红外图像的方法来获取更多的红外图像数据。首先,用现有的彩色图像和红外图像数据构建成对的数据集;然后,基于卷积神经网络、转置卷积神经网络构建生成对抗网络（GAN）模型的生成器和鉴别器;接着,基于成对的数据集来训练GAN模型,直到生成器和鉴别器之间达到纳什平衡状态;最后,用训练好的生成器将彩色图像从彩色域变换到红外域。基于定量评估标准对实验结果进行了评估,结果表明,所提方法可以生成高质量的红外图像,并且相较于在损失函数中不加正则化项,在损失函数中加入L1和L2正则化约束后,该方法的FID分数值平均分别降低了23.95和20.89。作为一种无监督的数据增强方法,该方法也可以被应用于其他缺少数据的目标识别、目标检测、数据不平衡等视觉任务中。     

Rendering the image of glare effect based on paired and unpaired dual generative adversarial network

It is a great challenge to rendering glare on image as the current rendering algorithms did not consider well the refraction of human eyes, thus the effect of rendering, in some critical application such as vehicle headlamps, is not real and may affect the safety evaluation. The traditional glare rendering algorithm relies on a large number of hand-designed wave optics processing operators, not only cannot complete the rendering work online in real time, but also cannot cope with the complex and changeable imaging conditions in reality. The mainstream generative adversarial network based algorithms in the field of image style translation are introduced to generate glare effect, which could be rendering online in a real time, however they still fail to render some effects such as detail distortion. In this work, we present a novel glare simulation generation method which is the first algorithm to apply a generative model based style transfer method to glare rendering. In a nutshell, a new method named Glare Generation Network is proposed to aggregate the benefits of content diversity and style consistency, which combines both paired and unpaired branch in a dual generative adversarial network. Our approach increase the structural similarity index measure by at least 0.039 on the custom darkroom vehicle headlamp dataset. We further show our method significantly improve the inference speed.     

融合残差密集块自注意力机制和生成对抗网络的对抗攻击防御模型

神经网络在图像分类任务上表现优异,但它极易受添加微小扰动的对抗样本的影响,输出错误的分类结果;而目前防御方法存在图像特征提取能力不足、对图像关键区域特征关注较少的问题.针对这些问题,提出了一种融合残差密集块(RDB)自注意力机制和生成对抗网络(GAN)的攻击防御模型——RD-SA-DefGAN.该模型将GAN和投影梯度...     

Machine fault diagnosis with small sample based on variational information constrained generative adversarial network

In actual engineering scenarios, limited fault data leads to insufficient model training and over-fitting, which negatively affects the diagnostic performance of intelligent diagnostic models. To solve the problem, this paper proposes a variational information constrained generative adversarial network (VICGAN) for effective machine fault diagnosis. Firstly, by incorporating the encoder into the discriminator to map the deep features, an improved generative adversarial network with stronger data synthesis capability is established. Secondly, to promote the stable training of the model and guarantee better convergence, a variational information constraint technique is utilized, which constrains the input signals and deep features of the discriminator using the information bottleneck method. In addition, a representation matching module is added to impose restrictions on the generator, avoiding the mode collapse problem and boosting the sample diversity. Two rolling bearing datasets are utilized to verify the effectiveness and stability of the presented network, which demonstrates that the presented network has an admirable ability in processing fault diagnosis with few samples, and performs better than state-of-the-art approaches.     

生成式对抗网络及其计算机视觉应用研究综述

目的 生成式对抗网络（GAN）的出现为计算机视觉应用提供了新的技术和手段,它以独特零和博弈与对抗训练的思想生成高质量的样本,具有比传统机器学习算法更强大的特征学习和特征表达能力。目前在机器视觉领域尤其是样本生成领域取得了显著的成功,是当前研究的热点方向之一。方法 以生成式对抗网络的不同模型及其在计算机视觉领域的应用为研究对象,在广泛调研文献特别是GAN的最新发展成果基础上,结合不同模型的对比试验,对每种方法的基本思想、方法特点及使用场景进行分析,并对GAN的优势与劣势进行总结,阐述了GAN研究的现状、在计算机视觉上的应用范围,归纳生成式对抗网络在高质量图像生成、风格迁移与图像翻译、文本与图像的相互生成和图像的还原与修复等多个计算机视觉领域的研究现状和发展趋势,并对每种应用的理论改进之处、优点、局限性及使用场景进行了总结,对未来可能的发展方向进行展望。结果 GAN的不同模型在生成样本质量与性能上各有优劣。当前的GAN模型在图像的处理上取得较大的成就,能生成以假乱真的样本,但是也存在网络不收敛、模型易崩溃、过于自由不可控的问题。结论 GAN作为一种新的生成模型具有很高的研究价值与应用价值,但目前存在一些理论上的桎梏亟待突破,在应用方面生成高质量的样本、逼真的场景是值得研究的方向。     

基于卷积自编码生成式对抗网络的高分辨率破损图像修复

目的 破损图像修复是一项具有挑战性的任务,其目的是根据破损图像中已知内容对破损区域进行填充。许多基于深度学习的破损图像修复方法对大面积破损的图像修复效果欠佳,且对高分辨率破损图像修复的研究也较少。对此,本文提出基于卷积自编码生成式对抗网络(convolutional auto-encoder generative adversarial network,CAE-GAN)的修复方法。方法 通过训练生成器学习从高斯噪声到低维特征矩阵的映射关系,再将生成器生成的特征矩阵升维成高分辨率图像,搜索与待修复图像完好部分相似的生成图像,并将对应部分覆盖到破损图像上,实现高分辨率破损图像的修复。结果 通过将学习难度较大的映射关系进行拆分,降低了单个映射关系的学习难度,提升了模型训练效果,在4个数据集上对不同破损程度的512×512×3高分辨率破损图像进行修复,结果表明,本文方法成功预测了大面积缺失区域的信息。与CE(context-encoders)方法相比,本文方法在破损面积大的图像上的修复效果提升显著,峰值信噪比(peak signal to noise ratio,PSNR)和结构相似性(str...