全像素双核成像技术及应用研究综述

全像素双核(dual-pixel, DP)自动对焦(dual-pixel CMOS auto focus, DP CMOS AF)采用混合检测自动对焦，其在每个像素配备两个光电二极管,使每个像素既参与对焦又参与成像，克服了传统相位检测自动对焦和反差检测自动对焦技术的缺点。根据离焦视差估计图像合焦镜头所需移动的距离，DP自动对焦具有更快的对焦速度和更高的对焦精度，因此广泛应用在手持设备中(如手机相机、单反相机等)。由于全像素双核传感器将每个像素分成两半，该传感器一次拍摄即可得到两幅图像。这两幅图像(全像素双核图像对)可以看做一个具有相同曝光时间和严格校正的小基线立体图像对。该图像对的视差与图像模糊程度相对应，只在离焦区域存在视差。全像素双核传感器不仅用于自动对焦，而且可以用于深度估计、散焦去模糊和反射去除等方面。本文系统地综述了全像素双核传感器的自动对焦、成像原理及研究现状，并进一步展望其未来发展。1)对自动对焦技术进行介绍，对比了传统对焦与全像素双核对焦；2)详细分析了全像素双核传感器的成像原理、成像模型及特点；3)系统地介绍了全像素双核在计算机视觉领域应用的最新进展，从深度估计、反射去除和离焦模糊去除等方面进行全面阐述及分析； 4)适当的数据集是基于深度学习方法的基础，对目前的全像素双核数据集进行了介绍；5)分析了全像素双核在计算机视觉领域面临的挑战与机遇，对未来的全像素双核应用方向进行了讨论与展望。

Dual-pixel imaging and applications: an overview

Dual-pixel (DP) sensor is a kind of Canon-originated hardware technique for autofocusing in 2013. Conventional autofocus methods are divided into two major categories: phase-based and contrast-based methods. However, phase-based autofocusing has higher electronic complexity, and contrast-based autofocusing runs slower in practice. Therefore, current hybrid detection autofocus technique is more concerned, which yields some pixels to imaging and focusing. However, the resolution loss issue cannot be avoided. Hybrid DP-based autofocusing methods enable each pixel to be integrated for both imaging and focusing, which improves the cost-efficiency focusing accuracy. Therefore, it has been widely used in mobile phone cameras and digital single lens reflex (DSLR) cameras. In recent years, DP sensors have been offered by sensor manufacturers and occupied the vast majority of camera sensor market. To guarantee the performance of focusing and imaging, each pixel in a DP-based sensor is equipped with two photodiodes. A DP-based sensor can assign each pixel to two halves and two images can be obtained simultaneously. These two images (DP image pair) could be viewed as a perfectly rectified stereo image pair in relation to a tiny baseline and the same exposure time, or as a two viewing angled light field camera. Unlike stereo image pairs, DP image pairs only have disparity at the out-of-focus regions, while the in-focus regions have no disparity. Defocus disparity, which only exists in the out-of-focus regions, is directly connected with the depth of the scene being captured and is generated by the point spread function. The point spread functions of the left and right views of DP are approximately symmetric, and the point spread functions based focus are relatively symmetric before and after as well. This relationship and the special point spread function can provide extra information for various computer vision tasks. Therefore, the obtained DP image pair can also be used for depth estimation, defocus deblurring and reflection removal beyond automatic focusing applications of the DP sensors. In particular, the relationship between depth and blur size in DP sensor is effective to deal with the depth from defocus task and the defocus deblur task. We critically review the autofocus, imaging principle and current situation of the DP-based sensor. 1) To provide a basic understanding of the dual-pixel sensors, we introduce the dual-pixel imaging model and imaging principle. 2) To specify the breakthrough of them, we carry out comparative analysis related to dual-pixel research in recent years. 3) To develop a reference for researchers further, we trace the current open-source dual-pixel datasets and simulators to facilitate data acquisition. Specifically, we firstly describe dual-pixel from the point of view of enabling automatic focus, where three conventional autofocus methods: 1) phase detection autofocus (PDAF), 2) contrast detection autofocus (CDAF), and 3) hybrid autofocus. The principle and priority of dual-pixel autofocus are critically reviewed in Section I. In Section II, we review the relevant optical concepts and camera imaging model. The imaging principle and geometric features of dual-pixel are introduced on four aspects: 1) dual-pixel geometry, 2) dual-pixel affine ambiguity, 3) dual-pixel point spread function, and 4) the difference between dual-pixel image pair and stereo image pair. It shows that DP image pairs can aid downstream tasks and how to mine effective hidden information from the DP image pairs. DP-based defocus disparity is linked to the contexted depth in terms of the affine ambiguity of dual-pixel, which can be used as a cue of depth estimation, defocus deblur and other related tasks. In Section III, we summarize the applications of the DP image pairs in the context of three computer vision tasks: 1) depth estimation, 2) reflection removal, and 3) defocus deblur. As appropriate datasets are fundamental to designing deep learning based architecture of neural networks better in contrast to conventional methods, we briefly introduce the community-derived DP datasets and summarize the algorithm principles of the current DP simulators. Finally, the future challenges and opportunities of the DP sensor have been discussed further in Section V.