Zernike-moment-based image super resolution

Multiframe super-resolution (SR) reconstruction aims to produce a high-resolution (HR) image using a set of low-resolution (LR) images. In the process of reconstruction, fuzzy registration usually plays a critical role. It mainly focuses on the correlation between pixels of the candidate and the reference images to reconstruct each pixel by averaging all its neighboring pixels. Therefore, the fuzzy-registration-based SR performs well and has been widely applied in practice. However, if some objects appear or disappear among LR images or different angle rotations exist among them, the correlation between corresponding pixels becomes weak. Thus, it will be difficult to use LR images effectively in the process of SR reconstruction. Moreover, if the LR images are noised, the reconstruction quality will be affected seriously. To address or at least reduce these problems, this paper presents a novel SR method based on the Zernike moment, to make the most of possible details in each LR image for high-quality SR reconstruction. Experimental results show that the proposed method outperforms existing methods in terms of robustness and visual effects.     

Multi-objective super resolution: concepts and examples

Described methods for simultaneously generating the super-resolved depth map and the image from LR observations. Structural information is embedded within the observations and, through the two formulations of DFD and SFS problems, we were able to generate the super-resolved images and the structures. The first method described here avoids correspondence and warping problems inherent in current SR techniques involving the motion cue in the LR observations and uses a more natural depth-related defocus as a natural cue in real aperture imaging. The second method, while again avoiding the correspondence problems, also demonstrates the usefulness of the generalized interpolation scheme leading to more flexibility in the final SR image, in the sense that the LR image can be viewed at SR with an arbitrary light source position. The quality of the super-resolved depth and intensity maps has been found to be quite good. The MAP-MRF framework that was used in both methods models both the surface normal and the intensity field as separate MRFs, and this helps in regularizing the solution.     

Low-Resolution Face Recognition via Coupled Locality Preserving Mappings

Practical face recognition systems are sometimes confronted with low-resolution face images. Traditional two-step methods solve this problem through employing super-resolution (SR). However, these methods usually have limited performance because the target of SR is not absolutely consistent with that of face recognition. Moreover, time-consuming sophisticated SR algorithms are not suitable for real-time applications. To avoid these limitations, we propose a novel approach for LR face recognition without any SR preprocessing. Our method based on coupled mappings (CMs), projects the face images with different resolutions into a unified feature space which favors the task of classification. These CMs are learned through optimizing the objective function to minimize the difference between the correspondences (i.e., low-resolution image and its high-resolution counterpart). Inspired by locality preserving methods for dimensionality reduction, we introduce a penalty weighting matrix into our objective function. Our method significantly improves the recognition performance. Finally, we conduct experiments on publicly available databases to verify the efficacy of our algorithm.     

Text-image super-resolution through anchored neighborhood regression with multiple class-specific dictionaries

In the dictionary-based image super-resolution (SR) methods, the resolution of the input image is enhanced using a dictionary of low-resolution (LR) and high-resolution (HR) image patches. Typically, a single dictionary is learned from all the patches in the training set. Then, the input LR patch is super-resolved using its nearest LR patches and their corresponding HR patches in the dictionary. In this paper, we propose a text-image SR method using multiple class-specific dictionaries. Each dictionary is learned from the patches of images of a specific character in the training set. The input LR image is segmented into text lines and characters, and the characters are preliminarily classified. Likewise, overlapping patches are extracted from the input LR image. Then, each patch is super-resolved through the anchored neighborhood regression, using n class-specific dictionaries corresponding to the top-n classification results of the character containing the patch. The final HR image is generated by aggregating all the super-resolved patches. Our method achieves significant improvements in visual image quality and OCR accuracy, compared to the related dictionary-based SR methods. This confirms the effectiveness of applying the preliminary character classification results and multiple class-specific dictionaries in text-image SR.     

Simulation of tissue atrophy using a topology preserving transformation model

We propose a method to simulate atrophy and other similar volumetric change effects on medical images. Given a desired level of atrophy, we find a dense warping deformation that produces the corresponding levels of volumetric loss on the labeled tissue using an energy minimization strategy. Simulated results on a real brain image indicate that the method generates realistic images of tissue loss. The method does not make assumptions regarding the mechanics of tissue deformation, and provides a framework where a pre-specified pattern of atrophy can readily be simulated. Furthermore, it provides exact correspondences between images prior and posterior to the atrophy that can be used to evaluate provisional image registration and atrophy quantification algorithms.     

A nonlinear least square technique for simultaneous image registration and super-resolution.

This paper proposes a new algorithm to integrate image registration into image super-resolution (SR). Image SR is a process to reconstruct a high-resolution (HR) image by fusing multiple low-resolution (LR) images. A critical step in image SR is accurate registration of the LR images or, in other words, effective estimation of motion parameters. Conventional SR algorithms assume either the estimated motion parameters by existing registration methods to be error-free or the motion parameters are known a priori. This assumption, however, is impractical in many applications, as most existing registration algorithms still experience various degrees of errors, and the motion parameters among the LR images are generally unknown a priori. In view of this, this paper presents a new framework that performs simultaneous image registration and HR image reconstruction. As opposed to other current methods that treat image registration and HR reconstruction as disjoint processes, the new framework enables image registration and HR reconstruction to be estimated simultaneously and improved progressively. Further, unlike most algorithms that focus on the translational motion model, the proposed method adopts a more generic motion model that includes both translation as well as rotation. An iterative scheme is developed to solve the arising nonlinear least squares problem. Experimental results show that the proposed method is effective in performing image registration and SR for simulated as well as real-life images.     

Curvature preserving image super-resolution with gradient-consistency-anisotropic-regularization prior

Single image super-resolution (SR) often suffers from annoying interpolation artifacts such as blur, jagged edges, and ringing. In this paper, we aim to achieve artifact-free SR reconstruction from an input low resolution (LR) image using adaptive de-convolution and curvature refinement. To achieve this, we propose a curvature preserving image SR method based on a gradient-consistency-anisotropic-regularization (GCAR) prior. The gradient consistency term effectively suppresses visual artifacts such as ringing and preserves sharp edges in images while the anisotropic regularization term adaptively preserves the high frequency information according to the gradient magnitude. The complementary two terms are elaborately combined into the GCAR prior for the SR reconstruction. The GCAR prior is very effective in preserving image details and recovering high frequency information. Moreover, we use curvature refinement to remove jagged artifacts caused by aliasing due to decimation. The proposed method employs an effective feedback-control loop which contains adaptive de-convolution, re-convolution, pixel substitution, and curvature refinement. The GCAR prior is utilized in the adaptive de-convolution step. Extensive experiments on various test images demonstrate that the proposed method produces natural-looking and artifact-free SR results in terms of both visual quality and quantitative performance.     

Image super-resolution with sparse neighbor embedding

Until now, neighbor-embedding-based (NE) algorithms for super-resolution (SR) have carried out two independent processes to synthesize high-resolution (HR) image patches. In the first process, neighbor search is performed using the Euclidean distance metric, and in the second process, the optimal weights are determined by solving a constrained least squares problem. However, the separate processes are not optimal. In this paper, we propose a sparse neighbor selection scheme for SR reconstruction. We first predetermine a larger number of neighbors as potential candidates and develop an extended Robust-SL0 algorithm to simultaneously find the neighbors and to solve the reconstruction weights. Recognizing that the k-nearest neighbor (k-NN) for reconstruction should have similar local geometric structures based on clustering, we employ a local statistical feature, namely histograms of oriented gradients (HoG) of low-resolution (LR) image patches, to perform such clustering. By conveying local structural information of HoG in the synthesis stage, the k-NN of each LR input patch is adaptively chosen from their associated subset, which significantly improves the speed of synthesizing the HR image while preserving the quality of reconstruction. Experimental results suggest that the proposed method can achieve competitive SR quality compared with other state-of-the-art baselines.     

Video super resolution based on non-local regularization and reliable motion estimation

Video super-resolution (SR) is a process for reconstructing high-resolution (HR) images by utilizing complementary information among multiple low-resolution (LR) images. Accurate estimation of the motion among the LR images significantly affects the quality of the reconstructed HR image. In this paper, we analyze the possible reasons for the inaccuracy of motion estimation and then propose a multi-lateral filter to regularize the process of motion estimation. This filter can adaptively correct motion estimation according to the estimation reliability, image intensity discontinuity, and motion dissimilarity. Furthermore, we introduce a non-local prior to solve the ill-posed problem of HR image reconstruction. This prior can fully utilize the self-similarities existing in natural images to regularize the HR image reconstruction. Finally, we employ a Bayesian formulation to incorporate the two regularizations into one Maximum a Posteriori (MAP) estimation model, where the HR image and the motion estimation can be refined progressively in an alternative and iterative manner. In addition, an algorithm that estimates the blur kernel by analyzing edges in an image is also presented in this paper. Experimental results demonstrate that the proposed approaches are highly effective and compare favorably to state-of-the-art SR algorithms.     

Cross-view information interaction and feedback network for face hallucination

Hallucinating a photo-realistic frontal face image from a low-resolution (LR) non-frontal face image is beneficial for a series of face-related applications. However, previous efforts either focus on super-resolving high-resolution (HR) face images from nearly frontal LR counterparts or frontalizing non-frontal HR faces. It is necessary to address all these challenges jointly for real-world face images in unconstrained environment. In this paper, we develop a novel Cross-view Information Interaction and Feedback Network (CVIFNet), which simultaneously handles the non-frontal LR face image super-resolution (SR) and frontalization in a unified framework and interacts them with each other to further improve their performance. Specifically, the CVIFNet is composed of two feedback sub-networks for frontal and profile face images. Considering the reliable correspondence between frontal and non-frontal face images can be crucial and contribute to face hallucination in a different manner, we design a cross-view information interaction module (CVIM) to aggregate HR representations of different views produced by the SR and frontalization processes to generate finer face hallucination results. Besides, since 3D rendered facial priors contain rich hierarchical features, such as low-level (e.g., sharp edge and illumination) and perception level (e.g., identity) information, we design an identity-preserving consistency loss based on 3D rendered facial priors, which can ensure that the high-frequency details of frontal face hallucination result are consistent with the profile. Extensive experiments demonstrate the effectiveness and advancement of CVIFNet.     

基于迭代交替优化的图像盲超分辨率重建

基于深度卷积神经网络的图像超分辨率重建算法通常假设低分辨率图像的降质是固定且已知的,如双3次下采样等,因此难以处理降质(如模糊核及噪声水平)未知的图像。针对此问题,该文提出联合估计模糊核、噪声水平和高分辨率图像,设计了一种基于迭代交替优化的图像盲超分辨率重建网络。在所提网络中,图像重建器以估计的模糊核和噪声水平作为先验信息,由低分辨率图像重建出高分辨率图像;同时,综合低分辨率图像和估计的高分辨率图像,模糊核及噪声水平估计器分别实现模糊核和噪声水平的估计。进一步地,该文提出对模糊核/噪声水平估计器及图像重建器进行迭代交替的端对端优化,以提高它们的兼容性并使其相互促进。实验结果表明,与IKC, DASR, MANet, DAN等现有算法相比,提出方法在常用公开测试集(Set5, Set14, B100, Urban100)及真实场景图像上都取得了更优的性能,能够更好地对降质未知的图像进行重建;同时,提出方法在参数量或处理效率上也有一定的优势。     

Superresolution video reconstruction with arbitrary samplinglattices and nonzero aperture time

Printing from an NTSC source and conversion of NTSC source material to high-definition television (HDTV) format are some of the applications that motivate superresolution (SR) image and video reconstruction from low-resolution (LR) and possibly blurred sources. Existing methods for SR image reconstruction are limited by the assumptions that the input LR images are sampled progressively, and that the aperture time of the camera is zero, thus ignoring the motion blur occurring during the aperture time. Because of the observed adverse effects of these assumptions for many common video sources, this paper proposes (i) a complete model of video acquisition with an arbitrary input sampling lattice and a nonzero aperture time, and (ii) an algorithm based on this model using the theory of projections onto convex sets to reconstruct SR still images or video from an LR time sequence of images. Experimental results with real video are provided, which clearly demonstrate that a significant increase in the image resolution can be achieved by taking the motion blurring into account especially when there exists large interframe motion.     

Multiframe super-resolution based on half-quadratic prior with artifacts suppress

The multiframe super-resolution (SR) technique aims to obtain a high-resolution (HR) image by using a set of observed low-resolution (LR) images. In the reconstruction process, artifacts may be possibly produced due to the noise, especially in presence of stronger noise. In order to suppress artifacts while preserving discontinuities of images, in this paper a multiframe SR method is proposed by involving the reconstruction properties of the half-quadratic prior model together with the quadratic prior model using a convex combination. Moreover, by analyzing local features of the underlined HR image, these two prior models are combined by using an automatically calculated weight function, making both smooth and discontinuous pixels handled properly. A variational Bayesian inference (VBF) based algorithm is designed to efficiently and effectively seek the solution of the proposed method. With the VBF framework, motion parameters and hyper-parameters are all determined automatically, leading to an unsupervised SR method. The efficiency of the hybrid prior model is demonstrated theoretically and practically, which shows that our SR method can obtain better results from LR images even with stronger noise. Extensive experiments on several visual data have demonstrated the efficacy and superior performance of the proposed algorithm, which can not only preserve image details but also suppress artifacts.     

Image Enhancement for Fluid Lens Camera Based on Color Correlation

The novel field of fluid lens cameras introduces unique image processing challenges. Intended for surgical applications, these fluid optics systems have a number of advantages over traditional glass lens systems. These advantages include improved miniaturization and no moving parts while zooming. However, the liquid medium creates two forms of image degradation: image distortion, which warps the image such that straight lines appear curved, and nonuniform color blur, which degrades the image such that certain color planes appear sharper than others. We propose the use of image processing techniques to reduce these degradations. To deal with image warping, we employ a conventional method that models the warping process as a degree-six polynomial in order to invert the effect. For image blur, we propose an adapted perfect reconstruction filter bank that uses high frequency sub-bands of sharp color planes to improve blurred color planes. The algorithm adjusts the number of levels in the decomposition and alters a prefilter based on crude knowledge of the blurring channel characteristics. While this paper primarily considers the use of a sharp green color plane to improve a blurred blue color plane, these methods can be applied to improve the red color plane as well, or more generally adapted to any system with high edge correlation between two images.      

自适应聚类的全耦合稀疏学习算法

在图像处理领域,基于稀疏表示理论的图像超分辨力算法、高低分辨力字典与稀疏编码之间的映射关系是其中的2个关键环节。由于丰富多样的图像类型,单一字典并不能很好地表示图像。而在稀疏编码之间的映射关系上,严格相等的约束关系也限制了图像重建的效果。针对上述两个方面,采用包容性更强的多个字典与约束条件更为宽松的全耦合稀疏关系进行图像的超分辨力重建。在图像非局部自相似性的基础上,进行多次自适应聚类;挑选出最优的聚类,通过全耦合稀疏学习的图像超分辨力算法,得到多个字典;最后,对输入的低分辨力图像进行分类重建,得到高分辨力图片。实验结果表明,在图像Leaves,Barbara,Room上,本文的聚类算法比原全耦合稀疏学习算法在峰值信噪比(PSNR)上分别提升了0.51 dB,0.21 dB,0.15 dB。     

Hybrid dual stream blender for wide baseline view synthesis

Free navigation of a scene requires warping some reference views to some desired target viewpoint and blending them to synthesize a virtual view. Convolutional Neural Networks (ConvNets) based methods can learn both the warping and blending tasks jointly. Such methods are often designed for moderate inter-camera baseline distance and larger kernels are required for warping if the baseline distance increases. Algorithmic methods can in principle deal with large baselines, however the synthesized view suffers from artifacts near disoccluded pixels. We present a hybrid approach where first, reference views are algorithmically warped to the target position and then are blended via a ConvNet. Preliminary view warping allows reducing the size of the convolutional kernels and thus the learnable parameters count. We propose a residual encoder–decoder for image blending with a Siamese encoder to further keep the parameters count low. We also contribute a hole inpainting algorithm to fill the disocclusions in the warped views. Our view synthesis experiments on real multiview sequences show better objective image quality than state-of-the-art methods due to fewer artifacts in the synthesized images.     

Deep edge map guided depth super resolution

Accurate edge reconstruction is critical for depth map super resolution (SR). Therefore, many traditional SR methods utilize edge maps to guide depth SR. However, it is difficult to predict accurate edge maps from low resolution (LR) depth maps. In this paper, we propose a deep edge map guided depth SR method, which includes an edge prediction subnetwork and an SR subnetwork. The edge prediction subnetwork takes advantage of the hierarchical representation of color and depth images to produce accurate edge maps, which promote the performance of SR subnetwork. The SR subnetwork is a disentangling cascaded network to progressively upsample SR result, where every level is made up of a weight sharing module and an adaptive module. The weight sharing module extracts the general features in different levels, while the adaptive module transfers the general features to the specific features to adapt to different degraded inputs. Quantitative and qualitative evaluations on various datasets with different magnification factors demonstrate the effectiveness and promising performance of the proposed method. In addition, we construct a benchmark dataset captured by Kinect-v2 to facilitate research on real-world depth map SR.     

Super-resolution image reconstruction: a technical overview

A new approach toward increasing spatial resolution is required to overcome the limitations of the sensors and optics manufacturing technology. One promising approach is to use signal processing techniques to obtain an high-resolution (HR) image (or sequence) from observed multiple low-resolution (LR) images. Such a resolution enhancement approach has been one of the most active research areas, and it is called super resolution (SR) (or HR) image reconstruction or simply resolution enhancement. In this article, we use the term "SR image reconstruction" to refer to a signal processing approach toward resolution enhancement because the term "super" in "super resolution" represents very well the characteristics of the technique overcoming the inherent resolution limitation of LR imaging systems. The major advantage of the signal processing approach is that it may cost less and the existing LR imaging systems can be still utilized. The SR image reconstruction is proved to be useful in many practical cases where multiple frames of the same scene can be obtained, including medical imaging, satellite imaging, and video applications. The goal of this article is to introduce the concept of SR algorithms to readers who are unfamiliar with this area and to provide a review for experts. To this purpose, we present the technical review of various existing SR methodologies which are often employed. Before presenting the review of existing SR algorithms, we first model the LR image acquisition process.     

Combining self-learning based super-resolution with denoising for noisy images

In this paper, we propose a new learning based joint Super-Resolution (SR) and denoising algorithm for noisy images. The individual processing of denoising and SR when super-resolving a noisy image has drawbacks such as noise amplification, blurring and SR performance reduction. In the proposed joint method, principal component analysis (PCA) based denoising is closely combined with a self-learning SR framework in order to minimize the SR visual quality degradation caused by noise. Experimental results show that the joint method achieves an SR image quality improvement in terms of noise and blurring, when compared with the state-of-the-art joint method and sequential combinations of individual denoising and SR.