Self-supervised multi-scale pyramid fusion networks for realistic bokeh effect rendering

Images with visual pleasing bokeh effect are often unattainable for mobile cameras with compact optics and tiny sensors. To balance the aesthetic requirements on photo quality and expensive high-end SLR cameras, synthetic bokeh effect rendering has emerged as an attractive machine learning topic for engineering applications on imaging systems. However, most of bokeh rendering models either heavily relied on prior knowledge such as scene depth or were topic-irrelevant data-driven networks without task-specific knowledge, which restricted models’ training efficiency and testing accuracy. Since bokeh is closely related to a phenomenon called ”circle of confusion”, therefore, in this paper, following the principle of bokeh generation, a novel self-supervised multi-scale pyramid fusion network has been proposed for bokeh rendering. During the pyramid fusion process, structure consistencies are employed to emphasize the importance of respective bokeh components. Task-specific knowledge which mimics the ”circle of confusion” phenomenon through disk blur convolutions is utilized as self-supervised information for network training. The proposed network has been evaluated and compared with several state-of-the-art methods on a public large-scale bokeh dataset- the ”EBB!” Dataset. The experiment performance demonstrates that the proposed network has much better processing efficiency and can achieve better realistic bokeh effect with much less parameters size and running time. Related source codes and pre-trained models of the proposed model will be available soon on https://github.com/zfw-cv/MPFNet.     

Prediction-Based Object Tracking in Visual Sensor Networks

This paper is to investigate the mobile object tracking in visual sensor networks. When visual sensors equipped with cameras are randomly deployed in a monitoring environment, many sensors are involved in covering the same mobile object. In a visual sensor network, images of the object may be captured by different sensors in different orientations simultaneously, and the captured images are then sent back to a base station or server. However, achieving full coverage for a set of selected characteristic points of an object invariably involves a great deal of redundant image data consuming the transmission energy for a visual sensor network. A novel approach is proposed to overcome this problem. The minimal number of sensors required for set coverage can be determined by predicting the direction and speed of the mobile object. Such sets are capable of covering the maximal number of characteristic points of view related to the mobile object at one time. The simulation results show that this approach reduces transmission cost while preserving the maximal coverage range of mobile objects.     

基于色彩相关性的彩色图像清晰度评价算法

数字图像的清晰度评价对于相机自动对焦而言至关重要。针对现有清晰度评价函数的不足之处,提出了一种针对彩色数字图像的空域清晰度评价算法。此算法利用像素三刺激值之间的色差相关性判定像素点的色彩,建立清晰度色差因子;运用非线性函数提高了像素点的梯度增益系数,使该评价函数对特殊条件下图像的清晰度判定更敏感。实验证明,该评价函数在评价一般自然图像清晰度时比传统评价算法更为敏感;在典型评价函数评价特殊目标图像失效时,该算法依然具有良好的鲁棒性;此算法计算量较小,执行效率高,易于硬件实现。     

A fuzzy convolutional neural network for enhancing multi-focus image fusion

The images captured by the cameras contain distortions, misclassified pixels, uncertainties and poor contrast. Therefore, the multi-focus image fusion (MFIF) integrates various input image features to produce a single fused image using all its objects in focus. However, it is computationally complex, which leads to inconsistency. Hence, the MFIF method is employed to generate the fused image by integrating the fuzzy sets (FS) and convolutional neural network (CNN) to detect focused and unfocused parts in both source images. It is also compared with other competing six MFIF methods like Neutrosophic set based stationary wavelet transform (NSWT), guided filters, CNN, ensemble CNN, image fusion-based CNN and deep regression pair learning (DRPL). Benchmark datasets validate the superiority of the proposed FCNN method in terms of four non-reference assessment measures having mutual information (1.1678), edge information (0.7281), structural similarity (0.9850) and human perception (0.8020) and two reference metrics such as Peak signal-to-noise ratio (57.23) and root mean square error (1.814).     

采用光流估计的数字相机自动对焦算法

自动对焦技术对于数字相机至关重要，它是获取清晰图像的重要手段。针对复杂环境下多目标场景图像，提出了一种基于光流场估计的自动对焦算法。通过计算输入图像序列的光流场，对场景中的运动目标进行检测，根据目标运动属性准确判断出感兴趣目标。改进了Brenner清晰度评价方法，利用目标的二维边缘梯度信息建立评价函数，并且通过非线性增益提高评价函数的灵敏度，减小了噪声对评价值的影响。实验证明，该算法能够在主辅目标景深比达50倍的情况下分辨出感兴趣主目标，并在方差为0.02的随机噪声干扰下能有效地评价图像的清晰度；此算法将Brenner等评价函数的峰值稳定余量提高了1至4倍，对于不同图像具有良好的鲁棒性，易于硬件实现。     

Event-guided low light image enhancement via a dual branch GAN

In the low light conditions, images are corrupted by low contrast and severe noise, but event cameras capture event streams with clear edge structures. Therefore, we propose an Event-Guided Low Light Image Enhancement method using a dual branch generative adversarial networks and recover clear structure with the guide of events. To overcome the lack of paired training datasets, we first synthesize three datasets containing low-light event streams, low-light images, and the ground truth normal-light images. Then, in the generator network, we develop an end-to-end dual branch network consisting of a image enhancement branch and a gradient reconstruction branch. The image enhancement branch is employed to enhance the low light images, and the gradient reconstruction branch is utilized to learn the gradient from events. Moreover, we develops the attention based event-image feature fusion module which selectively fuses the event and low-light image features, and the fused features are concatenated into the image enhancement branch and gradient reconstruction branch, which respectively generate the enhanced images with clear structure and more accurate gradient images. Extensive experiments on synthetic and real datasets demonstrate that the proposed event guided low light image enhancement method produces visually more appealing enhancement images, and achieves a good performance in structure preservation and denoising over state-of-the-arts.     

A Robust In-Car Digital Image Stabilization Technique

Machine vision is a key technology used in an intelligent transportation system (ITS) to augment human drivers' visual capabilities. For the in-car applications, additional motion components are usually induced by disturbances such as the bumpy ride of the vehicle or the steering effect, and they will affect the image interpretation processes that is required by the motion field (motion vector) detection in the image. In this paper, a novel robust in-car digital image stabilization (DIS) technique is proposed to stably remove the unwanted shaking phenomena in the image sequences captured by in-car video cameras without the influence caused by moving object (front vehicles) in the image or intentional motion of the car, etc. In the motion estimation process, the representative point matching (RPM) module combined with the inverse triangle method is used to determine and extract reliable motion vectors in plain images that lack features or contain a large low-contrast area to increase the robustness in different imaging conditions, since most of the images captured by in-car video cameras include large low-contrast sky areas. An adaptive background evaluation model is developed to deal with irregular images that contain large moving objects (front vehicles) or a low-contrast area above the skyline. In the motion compensation processing, a compensating motion vector (CMV) estimation method with an inner feedback-loop integrator is proposed to stably remove the unwanted shaking phenomena in the images without losing the effective area of the images with a constant motion condition. The proposed DIS technique was applied to the on-road captured video sequences with various irregular conditions for performance demonstrations     

Depth image super-resolution algorithm based on structural features and non-local means

The resolution and quality of the depth map captured by depth cameras are limited due to sensor hardware limitations, which becomes a roadblock for further computer vision applications. In order to solve this problem, we propose a new method to enhance low-resolution depth maps using high-resolution color images. The structural-aware term is introduced because of the availability of structural information in color images and the assumption of identical structural features within local neighborhoods of color images and depth images captured from the same scene. We integrate the structural-aware term with color similarity and depth similarity within local neighborhoods to design a local weighting filter based on structural features. To use non-local self-similarity of images, the local weighting filter is combined with the concept of non-local means, and then a non-local weighting filter based on structural features is designed. Some experimental results show that super-resolution depth image can be reconstructed well by the process of the non-local filter and the local filter based on structural features. The proposed method can reconstruct much better high-resolution depth images compared with previously reported methods.     

A novel deep neural network for noise removal from underwater image

Underwater image processing technologies have always been challenging tasks due to the complex underwater environment. Images captured under water are not only affected by the water itself, but also by the diverse suspended particles that increase the effect of absorption and scattering. Moreover, these particles themselves are usually imaged on the picture, causing the spot noise signal to interfere with the target objects. To address this issue, we propose a novel deep neural network for removing the spot noise from underwater images. Its main idea is to train a generative adversarial network (GAN) to transform the noisy image to clean image. Based on the deep encoder and decoder framework, the skip connections are introduced to combine the features of low-level and high-level to help recover the original image. Meanwhile, the self-attention mechanism is employed to the generative network to capture global dependencies in the feature maps, which can generate the image with fine details at every location. Furthermore, we apply the spectral normalization to both the generative and discriminative networks to stabilize the training process. Experiments evaluated on synthetic and real-world images show that the proposed method outperforms many recent state-of-the-art methods in terms of quantitative and visual quality. Besides, the results also demonstrate that the proposed method has the good ability to remove the spot noise from underwater images while preserving sharp edge and fine details.     

Fusion of thermal and visible cameras for the application of pedestrian detection

In this paper, we propose methods to calibrate visible and thermal cameras and register their images in the application of pedestrian detection. We calibrate the camera using a checkerboard pattern mounted on a heated rig. We implement the image registration using three different approaches. In the first approach, we use the camera calibration information to generate control points from the checkerboard pattern. These control points are then used to register the images. In the second approach, we generate trajectory points for image registration using an external illuminated object. In the third approach, we achieve the registration through face tracking without the aid of any external object. The particle swarm optimization algorithm performs the image registration using the generated control and trajectory points, observed in both the cameras. We demonstrate the advantages of fusing the thermal and visible camera within a pedestrian detection algorithm. We evaluate the proposed registration algorithms and perform a comparison with baseline algorithms, i.e. genetic and simulated annealing algorithms. Additionally, we also perform a detailed parameter evaluation of the particle swarm optimization algorithm. The experimental results demonstrate the accuracy of the proposed algorithm and the advantages of thermal-visible camera fusion.     

A Novel Character Segmentation Method for Text Images Captured by Cameras

Due to the rapid development of mobile devices equipped with cameras, instant translation of any text seen in any context is possible. Mobile devices can serve as a translation tool by recognizing the texts presented in the captured scenes. Images captured by cameras will embed more external or unwanted effects which need not to be considered in traditional optical character recognition (OCR). In this paper, we segment a text image captured by mobile devices into individual single characters to facilitate OCR kernel processing. Before proceeding with character segmentation, text detection and text line construction need to be performed in advance. A novel character segmentation method which integrates touched character filters is employed on text images captured by cameras. In addition, periphery features are extracted from the segmented images of touched characters and fed as inputs to support vector machines to calculate the confident values. In our experiment, the accuracy rate of the proposed character segmentation system is 94.90%, which demonstrates the effectiveness of the proposed method.     

Virtual camera calibration and stereo correspondence of single-lens bi-prism stereovision system using geometrical approach

This paper proposes a simple geometrical ray approach to calibrate the extrinsic parameters of the virtual cameras and solve the stereo correspondence problem of the single-lens bi-prism stereovision system. Each images captured using this system can be divided into two sub-images which are generated by two virtual cameras due to the refraction through the bi-prism. This stereovision system is equivalent to the conventional two camera system and the two captured sub-images provide disparity which can be used for depth recovery. The virtual cameras will be calibrated geometrically and the correspondence problem of this system will be solved by applying epipolar geometry constraint on the generated virtual cameras instead of the real CCD camera. Experiments are conducted to validate the proposed method and the results are compared to the conventional approach to confirm its accuracy and effectiveness.     

家用摄录机的电子图像稳定系统

本文介绍采用模糊数学控制大规模数字处理电路的电子图像稳定新技术,克服手握一体化摄录机引起的图像晃动效应,以改善图像质量。该技术可广泛应用于视频处理各领域。     

A Hardware Acceleration Platform for Digital Holographic Imaging

This paper presents a hardware acceleration platform for image reconstruction in digital holographic imaging. The hardware accelerator executes a computationally demanding reconstruction algorithm which transforms an interference pattern captured on a digital image sensor into visible images. Focus in this work is to maximize computational efficiency, and to minimize the external memory transfer overhead, as well as required internal buffering. The paper presents an efficient processing datapath with a fast transpose unit and an interleaved memory storage scheme. The proposed architecture results in a speedup with a factor 3 compared with the traditional column/row approach for calculating the two-dimensional FFT. Memory sharing between the computational units reduces the on-chip memory requirements with over 50%. The custom hardware accelerator, extended with a microprocessor and a memory controller, has been implemented on a custom designed FPGA platform and integrated in a holographic microscope to reconstruct images. The proposed architecture targeting a 0.13 µm CMOS standard cell library achieves real-time image reconstruction with 20 frames per second.     

Noise reduction using multiscale bilateral decomposition for digital color images

In this paper, we propose a noise reduction algorithm for digital color images using a nonlinear image decomposition approach. Most existing noise reduction methods do not adequately consider spatial correlation of color noise in digital color images. Color noise components in color images captured by digital cameras are observed as irregular grains with various sizes and shapes, which are spatially randomly distributed. We use a modified multiscale bilateral decomposition to effectively separate signal and mixed-type noise components, in which a noisy input image is decomposed into a base layer and several detail layers. A base layer contains strong edges, and most of noise components are contained in detail layers. Noise components in detail layers are reduced by an adaptive thresholding function. We obtain a denoised image by combining a base layer and noise-reduced detail layers. Experimental results show the effectiveness of the proposed algorithm, in terms of both the peak signal-to-noise ratio and visual quality.     

Automatic panorama with auto-focusing based on image fusion for microscopic imaging system

It is usually desirable from a microscope imaging system to have an efficient auto-focusing and to maintain imaging quality throughout microscopy screening restricted automatically by the specimen borders. This paper presents a novel image fusion-based auto-focusing method and an automatic panorama confined with surroundings of the specimen so as to minimize the auto-scanning time for microscope imaging system. Multi-focus color image fusion is proposed to achieve the auto-focusing task for microscopic imaging. An image sequence is captured by using a microscope eyepiece camera with moving the microscope stage along Z-axis. Several images around a reference image are used to achieve in-focus image, instead of selecting a single image from the sequence. The reference image is an image given highest focus measurement value within the image sequence. Moreover, various evaluation criteria are utilized to analyze the performance of the proposed auto-focus approach on different color models for microscopic imaging. Microscope stage position along the Z-axis is automatically adjusted by image processing-based feedback system to maintain focus during scanning process. In this screening, the in-focus images with overlapped areas on the X–Y axes are stitched together to produce a mosaic image without any seams. In this process, the screening area is automatically constrained with the specimen regions which occupy 20–40 % of the glass surface. An artificial neural network-based learning algorithm is implemented to decide whether the specimen regions are within microscope objective field of view or not. The experimental studies of the proposed method were achieved on an image data set collected from the bright-field microscopy screening for Mycobacterium tuberculosis in specimen of Ziehl–Neelsen-stained sputum smears.     

自由立体显示拍摄系统中摄像机空间自由度的确定

自由立体显示器的立体显示效果与视差图的视差大小及性质密切相关,而视差的大小是受获取视差图时相机的摆放方式影响的。文章提出利用双目相机标定技术求解左右摄像机中心的空间位置关系以及两个摄像机光轴的空间夹角,并通过标定结果不断调节摄像机的位置,从而使两个摄像机光轴达到近似平行,此时获取的视差图就可以保证更好的立体显示效果。通过实验研究得到了两个相机空间夹角允许的变化范围。该研究结果可用于指导自由立体显示器拍摄系统的调节。     

Hyperspectral image super-resolution combining with deep learning and spectral unmixing

In recent years, hyperspectral image super-resolution has attracted the attention of many researchers and has become a hot topic in the field of computer vision. However, it is difficult to obtain high-resolution images due to imaging hardware devices. At present, many existing hyperspectral image super-resolution methods have not achieved good results. In this paper, we propose a hyperspectral image super-resolution method combining with deep residual convolutional neural network (DRCNN) and spectral unmixing. Firstly, the spatial resolution of the image is enhanced by learning a priori knowledge of natural images. The DRCNN reconstructs high spatial resolution hyperspectral images by concatenating multiple residual blocks, each containing two convolutional layers. Secondly, the spectral features of low-resolution and high-resolution hyperspectral images are linked by spectral unmixing. This approach aims to obtain the endmember matrix and the abundance matrix. The final reconstruction result is obtained by multiplying the endmember matrix and the abundance matrix. In addition, in order to improve the visual effect of the reconstructed image, the total variation regularity is used to impose constraints on the abundance matrix to enhance the relationship between the pixels. The experimental results of remote sensing data based on ground facts show that the proposed method has good performance and preserves spatial information and spectral information without the need for auxiliary images.     

A High Quality Depth Map Upsampling Method Robust to Misalignment of Depth and Color Boundaries

In recent years, fusion camera systems that consist of color cameras and Time-of-Flight (TOF) depth sensors have been popularly used due to its depth sensing capability at real-time frame rates. However, captured depth maps are limited in low resolution compared to the corresponding color images due to physical limitation of the TOF depth sensor. Most approaches to enhancing the resolution of captured depth maps depend on the implicit assumption that when neighboring pixels in the color image have similar values, they are also similar in depth. Although many algorithms have been proposed, they still yield erroneous results, especially when region boundaries in the depth map and the color image are not aligned. We therefore propose a novel kernel regression framework to generate the high quality depth map. Our proposed filter is based on the vector pointing similar pixels that represents the unit vector toward similar neighbors in the local region. The vectors are used to detect misaligned regions between color edges and depth edges. Unlike conventional kernel regression methods, our method properly handles misaligned regions by introducing the numerical analysis of the local structure into the kernel regression framework. Experimental comparisons with other data fusion techniques prove the superiority of the proposed algorithm.     

Design of a systolic-pipelined architecture for real-time enhancement of color video stream based on an illuminance-reflectance model

A high performance digital architecture for the implementation of a nonlinear image enhancement technique is proposed in this paper. The image enhancement is based on an illuminance-reflectance model which improves the visual quality of digital images and video captured under insufficient or non-uniform lighting conditions. The algorithm shows robust performance with appropriate dynamic range compression, good contrast, accurate and consistent color rendition. The algorithm contains a large number of complex computations and thus it requires specialized hardware implementation for real-time applications. Systolic, pipelined and parallel design techniques are utilized effectively in the proposed FPGA-based architectural design to achieve real-time performance. Approximation techniques are used in the hardware algorithmic design to achieve high throughput. The video enhancement system is implemented using Xilinx's multimedia development board that contains a VirtexII-X2000 FPGA and it is capable of processing approximately 63 Mega-pixels (Mpixels) per second.