A nonlinear least square technique for simultaneous image registration and super-resolution.

This paper proposes a new algorithm to integrate image registration into image super-resolution (SR). Image SR is a process to reconstruct a high-resolution (HR) image by fusing multiple low-resolution (LR) images. A critical step in image SR is accurate registration of the LR images or, in other words, effective estimation of motion parameters. Conventional SR algorithms assume either the estimated motion parameters by existing registration methods to be error-free or the motion parameters are known a priori. This assumption, however, is impractical in many applications, as most existing registration algorithms still experience various degrees of errors, and the motion parameters among the LR images are generally unknown a priori. In view of this, this paper presents a new framework that performs simultaneous image registration and HR image reconstruction. As opposed to other current methods that treat image registration and HR reconstruction as disjoint processes, the new framework enables image registration and HR reconstruction to be estimated simultaneously and improved progressively. Further, unlike most algorithms that focus on the translational motion model, the proposed method adopts a more generic motion model that includes both translation as well as rotation. An iterative scheme is developed to solve the arising nonlinear least squares problem. Experimental results show that the proposed method is effective in performing image registration and SR for simulated as well as real-life images.     

自适应聚类的全耦合稀疏学习算法

在图像处理领域,基于稀疏表示理论的图像超分辨力算法、高低分辨力字典与稀疏编码之间的映射关系是其中的2个关键环节。由于丰富多样的图像类型,单一字典并不能很好地表示图像。而在稀疏编码之间的映射关系上,严格相等的约束关系也限制了图像重建的效果。针对上述两个方面,采用包容性更强的多个字典与约束条件更为宽松的全耦合稀疏关系进行图像的超分辨力重建。在图像非局部自相似性的基础上,进行多次自适应聚类;挑选出最优的聚类,通过全耦合稀疏学习的图像超分辨力算法,得到多个字典;最后,对输入的低分辨力图像进行分类重建,得到高分辨力图片。实验结果表明,在图像Leaves,Barbara,Room上,本文的聚类算法比原全耦合稀疏学习算法在峰值信噪比(PSNR)上分别提升了0.51 dB,0.21 dB,0.15 dB。     

Depth map super-resolution via low-resolution depth guided joint trilateral up-sampling

In this paper, a new method is proposed to address the depth map super resolution (SR) and denoising problems simultaneously. Unlike the existing methods, the proposed approach uses LR depth map as a guidance in each filtering iteration during the whole process to fully exploit the geometric information in it. A joint trilateral upsampling model is proposed to fuse the projected spatial distance measured from the LR depth map, the intensity variance extracted from the associated color image and the HR depth map generated in the last iteration to refine the HR depth map iteratively. Compared with the existing approaches, the proposed approach presents superior results in avoiding texture copying artifacts as misalignments existing between the depth map and color image. Also, for the depth with noises, it can provide stronger de-noising effects with much clearer edges in the processed results. On average, it only requires 7.67 iterations to reach convergence, which is very efficient and outperforms the representative approaches in terms of computational complexity, objective quality and subjective quality.     

Specification of the observation model for regularized image up-sampling.

Regularization is one of the most promising methods for image up-sampling, which is an ill-posed inverse problem. A key element of such a regularization approach is the observation model relating the observed lower resolution (LR) image to the desired higher resolution (HR) up-sampled image, used in the data-fidelity term of the regularization cost function. This paper presents an algorithm to determine this observation model based on a model of the physical acquisition process for the LR image, and the ideal acquisition process for the desired HR image, both from the same underlying continuous image. The method is illustrated with typical scenarios corresponding to LR and HR cameras modeled by either Gaussian or rectangular apertures. Experiments with some regularized image up-samplers demonstrate the importance of using the correct, adapted observation model as determined by our algorithm. Index Terms-Camera aperture, data fidelity, image up-sampling, interpolation, multidimensional signal processing, observation model, power spectral density (PSD), super-resolution.     

Video super resolution based on non-local regularization and reliable motion estimation

Video super-resolution (SR) is a process for reconstructing high-resolution (HR) images by utilizing complementary information among multiple low-resolution (LR) images. Accurate estimation of the motion among the LR images significantly affects the quality of the reconstructed HR image. In this paper, we analyze the possible reasons for the inaccuracy of motion estimation and then propose a multi-lateral filter to regularize the process of motion estimation. This filter can adaptively correct motion estimation according to the estimation reliability, image intensity discontinuity, and motion dissimilarity. Furthermore, we introduce a non-local prior to solve the ill-posed problem of HR image reconstruction. This prior can fully utilize the self-similarities existing in natural images to regularize the HR image reconstruction. Finally, we employ a Bayesian formulation to incorporate the two regularizations into one Maximum a Posteriori (MAP) estimation model, where the HR image and the motion estimation can be refined progressively in an alternative and iterative manner. In addition, an algorithm that estimates the blur kernel by analyzing edges in an image is also presented in this paper. Experimental results demonstrate that the proposed approaches are highly effective and compare favorably to state-of-the-art SR algorithms.     

基于关键点逐层重建的人脸图像超分辨率方法

本文提出一种通过基于关键点逐层重建的人脸图像超分辨率方法。该方法考虑到五官和眉毛局部部位的细节对超分重建的重要意义,本文提出对人脸关键点附近局部区域分别训练超分映射函数,并采用逐层迭代重建实现人脸超分的方法,减小直接重建目标图像的难度。针对超分映射函数,本文采取了线性和非线性两种学习方法,其中线性方法采用主成分分析(PCA),非线性方法采用自编码网络(AutoEncoder)。在超分重建阶段,先采用双线性插值作为初始化,进而利用学习得到的超分映射函数计算局部人脸图像超分,叠加到全局人脸图像,实现整体超分。基于关键点的人脸超分辨率图像质量较其他超分方法在五官的细节上有更好的效果,本文提出的方法在实验数据集上展现了良好的超分结果,验证了低分辨率证件照情况下的人脸识别的有效性。      

基于聚类字典的图像超分辨率重构算法

针对图像超分辨率(SR)重构在空间邻域选取过程中 细节特征易被大幅度特征分量淹没的问题,提出 一种基于聚类字典的SR重构(DD-NE)算法。图像SR重构是利用信号处理方 法来提高图像分辨 率,针对NE算法在空间邻域选取时细节信号易被大幅度信号淹没的问题,对输入图像及邻域 利用聚类字典进行 稀疏分解。从大、小幅值表示系数中分别重构大、小幅度特征子图,保护邻域计算中的小幅 度特征,并将 低分辨率(LR)图像库及输入图像使用聚类字典表示。细节信号以字典原子的形式得到表达 ,空间邻域度 量转换为字典原子间的度量,从而细节特征对邻域的选择更加准确。实验结果表明,相对于 NE算法,本文算法图像SR 重构的峰值信噪比(PSNR)值平均提升了1.1dB,有效改善了重构效果;重构时间仅为NE算法的30.9%。     

Blind single-image super resolution based on compressive sensing

Blind super resolution is an interesting area in image processing that can restore high resolution (HR) image without requiring prior information of the volatile point spread function (PSF). In this paper, a novel framework is proposed for blind single-image super resolution (SISR) problem based on compressive sensing (CS) framework that is one of the first works that considers general PSFs. The fundamental idea in the proposed approach is to use sparsity on a known sparse transform domain as a powerful regularizer in both the image and blur domains. Therefore, a new cost function with respect to the unknown HR image patch and PSF kernel is presented and minimization is performed based on two subproblems that are modeled similar to that of CS. Simulation results demonstrate the effectiveness of the proposed algorithm that is competitive with methods that use multiple LR images to achieve a single HR image.     

压缩视频超分辨率重建技术

超分辨率图像重建就是由低分辨率图像序列来估计高分辨率图像,而压缩视频的重建正成为当前研究的热点。本文首先分析了压缩视频重建的基础,建立了高、低分辨率图像间的关系,给出了量化噪声和运动矢量的模型;接着对目前最具有代表性的最大后验概率(MAP)算法、凸集投影(POCS)算法和迭代反向投影(IBP)算法进行了详细的阐述,并分别给出了实验结果;然后分析了运算的复杂度,介绍了几种实时化方法;最后针对目前存在的问题进行了展望,指出降质模型、运动估计、重建算法和实时应用将是今后研究的重点。     

多值压缩编码孔径超分辨率成像方法

针对近期提出的基于压缩感知(CS,compressed sensing)理论的压缩编码成像方法在重建后引入较多类似于噪声的伪影(artitacts)问题,为了使压缩编码成像方法获得更好成像质量的图像,本文提出一种改进的压缩编码成像方法。本文方法将多值模板(MVM)代替二值模板来增强编码质量,并利用自适应全变分(TV,total variation)去噪方法去除重建后的高分辨率图像的伪影。实验结果表明,这种方法很好地改进了压缩编码孔径(CCA)的成像质量,并且大幅提高了图像的信噪比(SNR)。     

一种Enlighten-GAN网络的指纹超分辨率重建方法北大核心CSCD

将低分辨率(LR)图像重建为高分辨率(HR)图像的主流模型是生成对抗网络(GAN)。然而,由于基于GAN的方法利用从其他图像中学习到的内容来恢复高频信息,在处理新的图像时往往会产生伪影。由于,指纹图像的特征比自然图像更加复杂。因此,将以前的网络应用于指纹图像,尤其是中等分辨率的图像,会导致收敛不稳定伪影效果更加严重。针对以上弊端,本文提出了一种Enlighten-GAN超分辨率方法,来解决指纹图像的重建问题。具体来说,我们设计了启发块来控制网络收敛到一个可靠的点,并利用自我监督分层感知损失以改进损失函数提升网络性能。实验结果证明Enlighten-GAN方法在指纹图像的重建效果性能上具有更加卓越的效果。     

Video super-resolution reconstruction based on deep convolutional neural network and spatio-temporal similarity

Existing learning-based super-resolution (SR) reconstruction algorithms are mainly designed for single image, which ignore the spatio-temporal relationship between video frames. Aiming at applying the advantages of learning-based algorithms to video SR field, a novel video SR reconstruction algorithm based on deep convolutional neural network (CNN) and spatio-temporal similarity (STCNN-SR) was proposed in this paper. It is a deep learning method for video SR reconstruction, which considers not only the mapping relationship among associated low-resolution (LR) and high-resolution (HR) image blocks, but also the spatio-temporal non-local complementary and redundant information between adjacent low-resolution video frames. The reconstruction speed can be improved obviously with the pre-trained end-to-end reconstructed coefficients. Moreover, the performance of video SR will be further improved by the optimization process with spatio-temporal similarity. Experimental results demonstrated that the proposed algorithm achieves a competitive SR quality on both subjective and objective evaluations, when compared to other state-of-the-art algorithms.     

视频监控图像的超分辨率复原研究

超分辨率复原技术是一种可用于提高图像细节辨识能力的有效方法。其在视频监控领域可望得到广泛应用。超分辨率图像处理技术通过融合多帧相似的低分辨率图像达到提高图像细节的目的。从而降低对监控视频采集硬件与后端辅助处理系统的要求,提高对特定目标的辨析能力。本文重点介绍了在视频监控领域较为实用的凸集投影算法、最大后验概率估计算法、基于对象的超分辨率复原方法、基于示例学习与多类预测器的超分辨率复原方法。对以上超分辨率复原方法实现流程的优缺点与其在视频图像监控领域的应用方法进行了相应分析。分析了超分辨率视频监控图像复原常用的基于块匹配与光流的对象运动估计方法。对超分辨率复原重建图像质量的评估标准也进行了相应讨论。     

Image super-resolution with sparse neighbor embedding

Until now, neighbor-embedding-based (NE) algorithms for super-resolution (SR) have carried out two independent processes to synthesize high-resolution (HR) image patches. In the first process, neighbor search is performed using the Euclidean distance metric, and in the second process, the optimal weights are determined by solving a constrained least squares problem. However, the separate processes are not optimal. In this paper, we propose a sparse neighbor selection scheme for SR reconstruction. We first predetermine a larger number of neighbors as potential candidates and develop an extended Robust-SL0 algorithm to simultaneously find the neighbors and to solve the reconstruction weights. Recognizing that the k-nearest neighbor (k-NN) for reconstruction should have similar local geometric structures based on clustering, we employ a local statistical feature, namely histograms of oriented gradients (HoG) of low-resolution (LR) image patches, to perform such clustering. By conveying local structural information of HoG in the synthesis stage, the k-NN of each LR input patch is adaptively chosen from their associated subset, which significantly improves the speed of synthesizing the HR image while preserving the quality of reconstruction. Experimental results suggest that the proposed method can achieve competitive SR quality compared with other state-of-the-art baselines.     

Zernike-moment-based image super resolution

Multiframe super-resolution (SR) reconstruction aims to produce a high-resolution (HR) image using a set of low-resolution (LR) images. In the process of reconstruction, fuzzy registration usually plays a critical role. It mainly focuses on the correlation between pixels of the candidate and the reference images to reconstruct each pixel by averaging all its neighboring pixels. Therefore, the fuzzy-registration-based SR performs well and has been widely applied in practice. However, if some objects appear or disappear among LR images or different angle rotations exist among them, the correlation between corresponding pixels becomes weak. Thus, it will be difficult to use LR images effectively in the process of SR reconstruction. Moreover, if the LR images are noised, the reconstruction quality will be affected seriously. To address or at least reduce these problems, this paper presents a novel SR method based on the Zernike moment, to make the most of possible details in each LR image for high-quality SR reconstruction. Experimental results show that the proposed method outperforms existing methods in terms of robustness and visual effects.     

Text-image super-resolution through anchored neighborhood regression with multiple class-specific dictionaries

In the dictionary-based image super-resolution (SR) methods, the resolution of the input image is enhanced using a dictionary of low-resolution (LR) and high-resolution (HR) image patches. Typically, a single dictionary is learned from all the patches in the training set. Then, the input LR patch is super-resolved using its nearest LR patches and their corresponding HR patches in the dictionary. In this paper, we propose a text-image SR method using multiple class-specific dictionaries. Each dictionary is learned from the patches of images of a specific character in the training set. The input LR image is segmented into text lines and characters, and the characters are preliminarily classified. Likewise, overlapping patches are extracted from the input LR image. Then, each patch is super-resolved through the anchored neighborhood regression, using n class-specific dictionaries corresponding to the top-n classification results of the character containing the patch. The final HR image is generated by aggregating all the super-resolved patches. Our method achieves significant improvements in visual image quality and OCR accuracy, compared to the related dictionary-based SR methods. This confirms the effectiveness of applying the preliminary character classification results and multiple class-specific dictionaries in text-image SR.     

Joint learning for single-image super-resolution via a coupled constraint

The neighbor-embedding (NE) algorithm for single-image super-resolution (SR) reconstruction assumes that the feature spaces of low-resolution (LR) and high-resolution (HR) patches are locally isometric. However, this is not true for SR because of one-to-many mappings between LR and HR patches. To overcome or at least to reduce the problem for NE-based SR reconstruction, we apply a joint learning technique to train two projection matrices simultaneously and to map the original LR and HR feature spaces onto a unified feature subspace. Subsequently, the k -nearest neighbor selection of the input LR image patches is conducted in the unified feature subspace to estimate the reconstruction weights. To handle a large number of samples, joint learning locally exploits a coupled constraint by linking the LR-HR counterparts together with the K-nearest grouping patch pairs. In order to refine further the initial SR estimate, we impose a global reconstruction constraint on the SR outcome based on the maximum a posteriori framework. Preliminary experiments suggest that the proposed algorithm outperforms NE-related baselines.     

基于迭代交替优化的图像盲超分辨率重建

基于深度卷积神经网络的图像超分辨率重建算法通常假设低分辨率图像的降质是固定且已知的,如双3次下采样等,因此难以处理降质(如模糊核及噪声水平)未知的图像。针对此问题,该文提出联合估计模糊核、噪声水平和高分辨率图像,设计了一种基于迭代交替优化的图像盲超分辨率重建网络。在所提网络中,图像重建器以估计的模糊核和噪声水平作为先验信息,由低分辨率图像重建出高分辨率图像;同时,综合低分辨率图像和估计的高分辨率图像,模糊核及噪声水平估计器分别实现模糊核和噪声水平的估计。进一步地,该文提出对模糊核/噪声水平估计器及图像重建器进行迭代交替的端对端优化,以提高它们的兼容性并使其相互促进。实验结果表明,与IKC, DASR, MANet, DAN等现有算法相比,提出方法在常用公开测试集(Set5, Set14, B100, Urban100)及真实场景图像上都取得了更优的性能,能够更好地对降质未知的图像进行重建;同时,提出方法在参数量或处理效率上也有一定的优势。     

Generative bayesian image super resolution with natural image prior

We propose a new single image super resolution (SR) algorithm via Bayesian modeling with a natural image prior modeled by a high-order Markov random field (MRF). SR is one of the long-standing and active topics in image processing community. It is of great use in many practical applications, such as astronomical observation, medical imaging, and the adaptation of low-resolution contents onto high-resolution displays. One category of the conventional approaches for image SR is formulating the problem with Bayesian modeling techniques and then obtaining its maximum-a-posteriori solution, which actually boils down to a regularized regression task. Although straightforward, this approach cannot exploit the full potential offered by the probabilistic modeling, as only the posterior mode is sought. On the other hand, current Bayesian SR approaches using the posterior mean estimation typically use very simple prior models for natural images to ensure the computational tractability. In this paper, we present a Bayesian image SR approach with a flexible high-order MRF model as the prior for natural images. The minimum mean square error (MMSE) criteria are used for estimating the HR image. A Markov chain Monte Carlo-based sampling algorithm is presented for obtaining the MMSE solution. The proposed method cannot only enjoy the benefits offered by the flexible prior, but also has the advantage of making use of the probabilistic modeling to perform a posterior mean estimation, thus is less sensitive to the local minima problem as the MAP solution. Experimental results indicate that the proposed method can generate competitive or better results than state-of-the-art SR algorithms.