联合多任务学习的人脸超分辨率重建

目的 人脸超分辨率重建是特定应用领域的超分辨率问题,为了充分利用面部先验知识,提出一种基于多任务联合学习的深度人脸超分辨率重建算法。方法 首先使用残差学习和对称式跨层连接网络提取低分辨率人脸的多层次特征,根据不同任务的学习难易程度设置损失权重和损失阈值,对网络进行多属性联合学习训练。然后使用感知损失函数衡量HR（high-resolution）图像与SR（super-resolution）图像在语义层面的差距,并论证感知损失在提高人脸语义信息重建效果方面的有效性。最后对人脸属性数据集进行增强,在此基础上进行联合多任务学习,以获得视觉感知效果更加真实的超分辨率结果。结果 使用峰值信噪比（PSNR）和结构相似度（SSIM）两个客观评价标准对实验结果进行评价,并与其他主流方法进行对比。实验结果显示,在人脸属性数据集（CelebA）上,在放大8倍时,与通用超分辨率MemNet（persistent memory network）算法和人脸超分辨率FSRNet（end-to-end learning face super-resolution network）算法相比,本文算法的PSNR分别提升约2.15 dB和1.2 dB。结论 实验数据与效果图表明本文算法可以更好地利用人脸先验知识,产生在视觉感知上更加真实和清晰的人脸边缘和纹理细节。     

数据外补偿的深度网络超分辨率重建

单张图像超分辨率重建受到多对一映射的困扰.对于给定的低分辨率图像块,存在若干高分辨率图像块与之对应.基于学习的方法受此影响,学习到的逆映射规则只能预测这些高分辨率图像块的均值,从而产生视觉上模糊的超分辨率重建结果.为了克服歧义性造成的高频细节损失,本文提出了一种基于深度网络,利用在线检索的数据进行高频信息补偿的图像超分辨率重建算法.该方法构建一个深度网络,通过三个分支预测高分辨率重建结果：一条旁路直接将输入的低分辨率图像输入到网络的最后一层;一条内部高频信息重建路径基于低分辨率图像回归预测高分辨率图像,重建高分辨率图像的主要结构;另一条外部高频信息补偿路径根据内部重建的结果,从在线检索到的相似图像中提取高频细节,对内部重建的重建结果进行细节补偿.在第二条路径中,为了有效提取高频信号并使之适应于内部重建的重建结构,本文在多层特征的测量和约束下,进行高频细节迁移.相比于之前基于云数据库的传统图像超分辨率方法,本文提出的方法是端对端可训练的（end-to-end trainable）,因此通过在大数据上进行学习,方法能同时建模内部重建和外部补偿,并能自动权衡两者利弊而给出最优的重建结果.图像超分辨率重建的实验结果表明,相比于最新的超分辨率算法,本文方法在主客观评价中均取得了更加优越的性能.     

Hyperspectral image super-resolution through clustering-based sparse representation

Promoting the spatial resolution of hyperspectral sensors is expected to improve computer vision tasks. However, due to the physical limitations of imaging sensors, the hyperspectral image is often of low spatial resolution. In this paper, we propose a new hyperspectral image super-resolution method from a low-resolution (LR) hyperspectral image and a high resolution (HR) multispectral image of the same scene. The reconstruction of HR hyperspectral image is formulated as a joint estimation of the hyperspectral dictionary and the sparse codes based on the spatial-spectral sparsity of the hyperspectral image. The hyperspectral dictionary is learned from the LR hyperspectral image. The sparse codes with respect to the learned dictionary are estimated from LR hyperspectral image and the corresponding HR multispectral image. To improve the accuracy, both spectral dictionary learning and sparse coefficients estimation exploit the spatial correlation of the HR hyperspectral image. Experiments show that the proposed method outperforms several state-of-art hyperspectral image super-resolution methods in objective quality metrics and visual performance.

     

基于半耦合稀疏表达的极低分辨率人脸识别

现有基于学习的人脸超分辨率算法假设高低分辨率特征具有流形一致性（耦合字典学习）,然而低分辨率图像的降质过程使得高低分辨率特征产生了“一对多”的映射关系偏差,减少了极低分辨率图像特征的判决信息,降低了超分辨率重建图像的识别率。针对这一问题,引入了半耦合稀疏字典学习模型,松弛高低分辨率流形一致性假设,同时学习稀疏表达字典和稀疏表达系数之间的映射函数,提升高低分辨率判决特征的一致性,在此基础上,引入协同分类模型,实现半耦合特征的高效分类。实验表明：相比于传统稀疏表达分类算法,算法不仅提高了识别率,并且还大幅度降低了时间开销,验证了半耦合稀疏学习字典在人脸识别中的有效性。     

Locally affine patch mapping and global refinement for image super-resolution

This paper deals with the super-resolution (SR) problem based on a single low-resolution (LR) image. Inspired by the local tangent space alignment algorithm in [16] for nonlinear dimensionality reduction of manifolds, we propose a novel patch-learning method using locally affine patch mapping (LAPM) to solve the SR problem. This approach maps the patch manifold of low-resolution image to the patch manifold of the corresponding high-resolution (HR) image. This patch mapping is learned by a training set of pairs of LR/HR images, utilizing the affine equivalence between the local low-dimensional coordinates of the two manifolds. The latent HR image of the input (an LR image) is estimated by the HR patches which are generated by the proposed patch mapping on the LR patches of the input. We also give a simple analysis of the reconstruction errors of the algorithm LAPM. Furthermore we propose a global refinement technique to improve the estimated HR image. Numerical results are given to show the efficiency of our proposed methods by comparing these methods with other existing algorithms.     

利用子空间稀疏特征的超分辨率图像重建算法

在传统超分辨率图像重建算法中,图像的梯度、纹理结构等特征通常是由人工设计的规则提取的,对于结构复杂、内容丰富的图像,这样提取到的特征不能精确地表达图像的全部信息,对图像的边缘和局部细节信息会造成缺失。而且在图像训练过程中,还会出现低分辨率[(LR)]和高分辨率[(HR)]图像特征图数量不一致、特征匹配度较低的问题。因此,如何提取表达能力更强的特征作为源图像的精确表示和训练过程中提高图像特征匹配度对图像的超分辨率重建至关重要。针对上述问题,提出了一种基于[PCANet]模型的超分辨率图像重建算法。首先通过具有高斯内核函数的[PCANet]模型提取图像的深层次特征,并且加入稀疏优化算法,对输出的特征映射矩阵迭代优化,得到其最佳投影矩阵,有效提升了特征映射的鲁棒性。然后利用学习获得的LR滤波器将提取到的图像的深度学习特征分解为多个稀疏特征,使用[ADMM]算法和SA-ADMM算法迭代更新得到其最优解以后,结合[LR]图像的稀疏特征和映射函数估计出HR图像的稀疏特征表示,最后和相应的[HR]滤波器进行卷积求和得到最终的重建图像。实验结果表明,该方法使重建图像的细节信息更好地保留,图像的边缘纹理更加清晰,客观评价指标平均[PSNR]值提高了0.21?dB以上,有效提升了图像重建的质量。     

Single Image Super-Resolution Reconstruction Technique based on A Single Hybrid Dictionary

A new sparse domain approach is proposed in this paper to realize the single image super-resolution (SR) reconstruction based upon one single hybrid dictionary, which is deduced from the mixture of both the high resolution (HR) image patch samples and the low resolution (LR) ones. Moreover, a linear model is proposed to characterize the relationship between the sparse representations of both the HR image patches and the corresponding LR ones over the same hybrid dictionary. It is shown that, the requirement on the identical sparse representation of both HR and LR image patches over the corresponding HR dictionary and the LR dictionary can be relaxed. It is unveiled that, the use of one single hybrid dictionary can not only provide a more flexible framework to keep the similar sparse characteristics between the HR patches and the corresponding degenerated LR patches, but also to accommodate their differences. On this basis, the sparse domain based SR reconstruction problem is reformulated. Moreover, the proposed linear model between the sparse representations of both the HR patch and the corresponding LR patch over the same hybrid dictionary offers us a new method to interpret the image degeneration characteristics in sparse domain. Finally, practical experimental results are presented to test and verify the proposed SR approach.     

结合支持向量回归和图像自相似的单幅图像超分辨率算法

目的 基于学习的单幅图像超分辨率算法是借助实例训练库由一幅低分辨率图像产生高分辨率图像。提出一种基于图像块自相似性和对非线性映射拟合较好的支持向量回归模型的单幅超分辨率方法,该方法不需使用外部图像训练库。方法 首先根据输入的低分辨率图像建立图像金字塔及包含低/高分辨率图像块对的集合;然后在低/高分辨率图像块对的集合中寻找与输入低分辨率图像块的相似块,利用支持向量回归模型学习这些低分辨率相似块和其对应的高分辨率图像块的中心像素之间的映射关系,进而得到未知高分辨率图像块的中心像素。结果 为了验证本文设计算法的有效性,选取结构和纹理不同的7幅彩色高分辨率图像,对其进行高斯模糊的2倍下采样后所得的低分辨率图像进行超分辨率重构,与双三次插值、基于稀疏表示及基于支持向量回归这3个超分辨率方法重建的高分辨率图像进行比较,峰值信噪比平均依次提升了2.37 dB、0.70 dB和0.57 dB。结论 实验结果表明,本文设计的算法能够很好地实现图像的超分辨率重构,特别是对纹理结构相似度高的图像具有更好的重构效果。     

稀疏字典编码的超分辨率重建

基于学习的超分辨率方法通常根据低分辨率图像从样本库中选取若干特征相似的匹配对象,再使用优化算法进行超分辨率估计,但其结果受匹配对象的质量限制,并且匹配特征一般只选择图像的几何结构信息,匹配准确性较低.提出了稀疏字典编码的超分辨率模型,将高、低分辨率图像特征块统一进行稀疏编码,建立高、低分辨率图像的稀疏关联,同步实现匹配搜索和优化估计,突破了上述方法的限制.应用形态分量分析法提取图像的特征数据,提高了特征匹配的准确性,并同步实现超分辨率重建和降噪功能.优化方法采用稀疏K-SVD算法以提高稀疏字典编码的计算速度.采用自然图像进行实验与其他基于学习的超分辨率算法相比,重建所得到的图像质量更优.     

基于多级隐空间信息约束的噪声人脸超分辨率算法

为了实现强噪声和模糊干扰下的低清人脸图像重建,提出一种基于多级隐空间信息约束的噪声人脸超分辨率算法.首先设计一个用于人脸有效信息提取的特征蒸馏网络, 并通过统计性抗干扰模型和隐空间特征对比算法移除噪声等无效信息,构建一个具有高噪声鲁棒性的人脸信息提取模型;然后,设计人脸重建网络,该网络利用提取的人脸特征重建高清人脸图像; 最后,通过人脸身份嵌入模型和离散小波变换模型,分别从超球面身份度量空间和小波域进一步对重建人脸的身份信息和空间结构进行约束.实验结果表明,所提出的算法不仅能够有效去除高噪声环境下的人脸噪声,而且还能有效提升人脸图像分辨率,获得更高的峰值信噪比(peak signal-to-noise ratio,PSNR)和结构相似度(structural similarity index,SSIM),具有较好的实用性.     

Face hallucination based on sparse local-pixel structure

In this paper, we propose a face-hallucination method, namely face hallucination based on sparse local-pixel structure. In our framework, a high resolution (HR) face is estimated from a single frame low resolution (LR) face with the help of the facial dataset. Unlike many existing face-hallucination methods such as the from local-pixel structure to global image super-resolution method (LPS-GIS) and the super-resolution through neighbor embedding, where the prior models are learned by employing the least-square methods, our framework aims to shape the prior model using sparse representation. Then this learned prior model is employed to guide the reconstruction process. Experiments show that our framework is very flexible, and achieves a competitive or even superior performance in terms of both reconstruction error and visual quality. Our method still exhibits an impressive ability to generate plausible HR facial images based on their sparse local structures.     

基于残差的图像超分辨率重建

提出一种基于图像残差的超分辨率重建算法.以原高分辨率图像与插值放大后图像之间的图像残差与低分辨率图像样本特征作为样本对,对其进行K均值分类,并对每类样本对采用KSVD(K-singular value decomposition)方法进行训练获得高、低分辨率字典对,然后根据测试样本与类中心的欧氏距离选择字典对,以与测试样本相近的多个类别所重建的结果加权获得图像残差,并结合低分辨率图像的插值结果获得高分辨率图像.实验结果表明,提出的方法具有更高的重建质量,且采用训练样本分类和相近类别的重建结果的加权和有利于提高图像重建质量.     

基于非负邻域嵌入和非局部正则化的单帧图像超分辨率重建算法

单帧图像超分辨率重建是指利用一幅低分辨率图像,通过相应的算法来获取一幅高分辨率图像的技术。提出了一种基于 非负邻域嵌入和 非局部正则化 的单帧图像超分辨率重建算法,以弥补传统邻域嵌入算法的不足。在训练阶段,首先对低分辨率图像预放大2倍,以保证在放大倍数较大时,高、低分辨率图像块之间的邻域关系也能得到较好的保持;在重建阶段,使用非负邻域嵌入来有效地解决近邻数的选取问题;最后利用图像块的非局部相似性构造非局部正则项对重建结果进行修正。实验结果表明,相对于传统算法,本方法的重建结果纹理丰富、边缘清晰。     

基于对偶回归和注意力机制的图像超分辨率重建网络

针对单幅图像超分辨率(single image super-resolution, SISR)重建算法存在低分辨率图像(LR)到高分辨率图像(HR)的映射学习具有不适定性,深层神经网络收敛慢且缺乏对高频信息的学习能力以及在深层神经网络传播过程中图像特征信息存在丢失的问题.本文提出了基于对偶回归和残差注意力机制的图像超分辨率重建网络.首先,通过对偶回归约束映射空间.其次,融合通道和空间注意力机制构造了残差注意力模块(RCSAB),加快模型收敛速度的同时,有效增强了对高频信息的学习.最后,融入密集特征融合模块,增强了特征信息流动性.在Set5、Set14、BSD100、Urban100四种基准数据集上与目前主流的单幅图像超分辨率算法进行对比,实验结果表明该方法无论是在客观质量评价指标还是主观视觉效果均优于对比算法.     

Image super-resolution by estimating the enhancement weight of self example and external missing patches

Image super-resolution (SR) is the process of generating a high-resolution (HR) image using one or more low-resolution (LR) inputs. Many SR methods have been proposed, but generating the small-scale structure of an SR image remains a challenging task. We hence propose a single-image SR algorithm that combines the benefits of both internal and external SR methods. First, we estimate the enhancement weights of each LR-HR image patch pair. Next, we multiply each patch by the estimated enhancement weight to generate an initial SR patch. We then employ a method to recover the missing information from the high-resolution patches and create that missing information to generate a final SR image. We then employ iterative back-projection to further enhance visual quality. The method is compared qualitatively and quantitatively with several state-of-the-art methods, and the experimental results indicate that the proposed framework provides high contrast and better visual quality, particularly for non-smooth texture areas.     

A high quality single-image super-resolution algorithm based on linear Bayesian MAP estimation with sparsity prior

This paper proposes a novel single-image super-resolution algorithm based on linear Bayesian maximum a posteriori (MAP) estimation and sparse representation. Starting from constructing several probability distribution priors in representation vector, we develop a linear Bayesian MAP estimator to acquire the most probable high-resolution (HR) image behind the low-resolution (LR) observation. Our new algorithm involves three main steps: (1) obtaining an initial estimate of the HR image via bi-cubic interpolation algorithm, (2) performing sparse coding on the initial estimate to get the representation vector and its support, (3) using the MAP estimator to restore the desired representation vector and then reconstructing the HR output. Simulated results show that the proposed method can achieve a more competitive performance both in subjective visual quality and in peak-signal-to-noise ratio (PSNR) and structural similarity (SSIM) measures, compared with other state-of-the-art super-resolution methods.     

基于非线性最小二乘的图像自适应SR重建以及运动估计

图像超分辨率（SR）重建是利用数字信号处理技术由一系列低分辨率观测图像得到高分辨率图像。为了扩展SR技术的应用范围,提出了一种同时进行图像超分辨率重建和全局运动估计的方法。该方法首先基于最大后验概率(MAP)给出了图像SR重建和运动估计框架,该框架不仅考虑了前后两次迭代所得的HR图像差值对最终重建图像的影响,而且引入了不同LR图像对重建图像的重要性权值,使得算法具有自适应性;然后将总体框架转换为图像SR重建模型和运动估计模型;最后基于非线性最小二乘法对模型进行优化求解,得出了SR重建图像及其全局运动域。实验表明,该方法不仅图像重建效果良好,并有着良好的收敛性。     

Global consistency,local sparsity and pixel correlation: A unified framework for face hallucination

In this paper, a novel two-phase framework is presented to deal with the face hallucination problem. In the first phase, an initial high-resolution (HR) face image is produced in patch-wise. Each input low-resolution (LR) patch is represented as a linear combination of training patches and the corresponding HR patch is estimated by the same combination coefficients. Realizing that training patches similar with the input may provide more appropriate textures in the reconstruction, we regularize the combination coefficients by a weighted _?2-norm${?}_2\mathrm{-}\mathrm{norm}$ minimization term which enlarges the coefficients for relevant patches. The HR face image is then initialized by integrating all the HR patches. In the second phase, three regularization models are introduced to produce the final HR face image. Different from most previous approaches which consider global and local priors separately, the proposed algorithm incorporates the global reconstruction model, the local sparsity model and the pixel correlation model into a unified regularization framework. Initializing the regularization problem with the HR image obtained in the first phase, the final output HR image can be optimized through an iterative procedure. Experimental results show that the proposed algorithm achieves better performances in both reconstruction error and visual quality.     

Hallucinating face by position-patch

A novel face hallucination method is proposed in this paper for the reconstruction of a high-resolution face image from a low-resolution observation based on a set of high- and low-resolution training image pairs. Different from most of the established methods based on probabilistic or manifold learning models, the proposed method hallucinates the high-resolution image patch using the same position image patches of each training image. The optimal weights of the training image position-patches are estimated and the hallucinated patches are reconstructed using the same weights. The final high-resolution facial image is formed by integrating the hallucinated patches. The necessity of two-step framework or residue compensation and the differences between hallucination based on patch and global image are discussed. Experiments show that the proposed method without residue compensation generates higher-quality images and costs less computational time than some recent face image super-resolution (hallucination) techniques.     

Super-resolution of human face image using canonical correlation analysis

Super-resolution reconstruction of face image is the problem of reconstructing a high resolution face image from one or more low resolution face images. Assuming that high and low resolution images share similar intrinsic geometries, various recent super-resolution methods reconstruct high resolution images based on a weights determined from nearest neighbors in the local embedding of low resolution images. These methods suffer disadvantages from the finite number of samples and the nature of manifold learning techniques, and hence yield unrealistic reconstructed images.To address the problem, we apply canonical correlation analysis (CCA), which maximizes the correlation between the local neighbor relationships of high and low resolution images. We use it separately for reconstruction of global face appearance, and facial details. Experiments using a collection of frontal human faces show that the proposed algorithm improves reconstruction quality over existing state-of-the-art super-resolution algorithms, both visually, and using a quantitative peak signal-to-noise ratio assessment.