基于视差重映射的立体图像视觉舒适度提升

目的 针对人眼观看立体图像内容可能存在的视觉不舒适性，基于视差对立体图像视觉舒适度的影响，提出了一种结合全局线性和局部非线性视差重映射的立体图像视觉舒适度提升方法。方法 首先，考虑双目融合限制和视觉注意机制，分别结合空间频率和立体显著性因素提取立体图像的全局和局部视差统计特征，并利用支持向量回归构建客观的视觉舒适度预测模型作为控制视差重映射程度的约束；然后，通过构建的预测模型对输入的立体图像的视觉舒适性进行分析，就欠舒适的立体图像设计了一个两阶段的视差重映射策略，分别是视差范围的全局线性重映射和针对提取的潜在欠舒适区域内视差的局部非线性重映射；最后，根据重映射后的视差图绘制得到舒适度提升后的立体图像。结果 在IVY Lab立体图像舒适度测试库上的实验结果表明，相较于相关有代表性的视觉舒适度提升方法对于欠舒适立体图像的处理结果，所提出方法在保持整体场景立体感的同时，能更有效地提升立体图像的视觉舒适度。结论 所提出方法能够根据由不同的立体图像特征构建的视觉舒适度预测模型来自动实施全局线性和局部非线性视差重映射过程，达到既改善立体图像视觉舒适度、又尽量减少视差改变所导致的立体感削弱的目的，从而提升立体图像的整体3维体验。

Visual comfort enhancement for stereoscopic images based on disparity remapping

Objective At present, 3D videos have become extensively integrated into the daily lives of people due to the immersive visual experience that they provide to users. However, viewers can experience visual discomfort when watching 3D videos, and even suffer from eye fatigue, headache, nausea, and other symptoms due to defects in 3D imaging technology. Therefore, the study of visual comfort enhancement methods for stereoscopic images or videos is highly significant to improve stereoscopic display technology and provide users with higher-quality 3D vision service. The factors that can cause visual discomfort when people watch stereoscopic images or videos include the followings: vergence-accommodation conflict, excessive cross and non-cross disparities, disparity distribution, spatial frequency, mismatch between left and right images, and object movement. Vergence-accommodation conflict is the fundamental cause of visual discomfort. Binocular vergence-accommodation conflict is characterized by a large disparity that occurs in 3D space. If the disparity is outside the fusion range, then the viewer cannot fuse the left and right images into a stereoscopic image, and instead, will see an unclear crosstalk image, thereby resulting in severe visual fatigue. Disparity distribution is also one of the main factors that affect visual comfort. Excessive cross disparity is more likely to cause visual discomfort than excessive non-cross disparity. When the entire image is located in front of the screen, visual comfort will be lower compared with when the entire image is positioned behind the screen. The disparity distribution of an image is more concentrated on the zero-disparity plane, thereby making the image more comfortable to view. As dispersion decreases, viewing the image becomes more comfortable. Spatial frequency also influences visual comfort by affecting binocular fusion limit. An image with high spatial frequency causes a higher degree of visual discomfort than an image with low spatial frequency. Disparity adjustment is the main method that can enhance the visual comfort of stereoscopic images because the vergence-accommodation conflict caused by the increased disparity is the main factor that leads to visual discomfort. Disparity adjustment methods can be divided into two categories: disparity shifting and disparity scaling. A disparity shifting method adjusts disparity by shifting the zero-disparity plane of the original image, thereby keeping the disparity range unchanged. Although this method has low computational complexity, simultaneously ensuring maximum cross disparity and non-cross disparity within the comfort zone is difficult regardless of how disparity is moved when the original disparity range exceeds a certain range of comfortable viewing area. Thus, visual discomfort remains unavoidable in this case. By contrast, the disparity range of the original scene can be linearly or nonlinearly adjusted into the comfort area by using a disparity scaling method. In general, excessive vergence-accommodation conflict can be avoided effectively by reducing the disparity range of the scene. However, when a large-scale disparity reduction is performed, the overall perceived depth of the stereoscopic image is significantly decreased, and an unnatural visual effect occurs due to the limited range of the comfortable viewing area. A new visual comfort enhancement method for stereoscopic images is proposed by combining global linear and local nonlinear disparity remapping based on the effect of disparity on visual comfort. This method can prevent visual discomfort when viewing stereoscopic images; it also balances the improvement of visual comfort of stereoscopic images and the weakening of the 3D sense of scenes.Method First, an objective visual comfort assessment model is constructed to automatically predict the visual comfort of stereoscopic images and to judge the improvement of the visual comfort of stereoscopic images during disparity adjustment. On the one hand, when binocular fusion limitation is considered, the global visual comfort features of stereoscopic images are extracted by combining spatial frequency and disparity. On the other hand, we perform a disparity statistical analysis on stereoscopic significant regions and obtain local visual comfort features based on the hypothesis that the human eye tends to pay excessive attention to perceived salient regions. Support vector regression is adopted in this study to construct the objective visual comfort prediction model for stereoscopic images by establishing the mapping relationship between features and subjective scores. Then, the visual comfort of the input stereoscopic image is analyzed using the constructed prediction model. A two-stage disparity remapping strategy is designed for less-comfortable stereoscopic images. This strategy consists of the global linear adjustment of the disparity range and the local nonlinear adjustment of the disparity in the extracted potentially less-comfortable regions. The global disparity remapping of the input disparity map is performed during the first stage to adjust the uncomfortable stereoscopic images to a relatively comfortable degree. The global disparity linear iterative adjustment process is performed if the predicted visual comfort objective score is less than the preset threshold. Only the global features are applied at this point to construct the visual comfort prediction function. Local nonlinear disparity remapping is then performed during the second stage to further enhance the viewing comfort of the stereoscopic image and maintain the 3D sense of the scene. The disparity of the potentially less-comfortable regions extracted from the disparity map after global linear remapping is adjusted via nonlinear iteration until the predicted visual comfort objective score is higher than the preset target threshold. The visual comfort of the adjusted stereoscopic image is predicted in conjunction with global and local features at this point. Lastly, an updated comfortable stereoscopic image is reconstructed via a rendering technique according to the remapped disparity map. Result A subjective evaluation experiment is designed on the IVY Lab stereoscopic image database to verify the effectiveness of the proposed method in improving the visual comfort and maintaining the 3D sense of stereoscopic images. Experimental results show that the proposed method can more effectively enhance the visual comfort of less-comfortable stereoscopic images while maintaining the 3D sense of scenes compared with state-of-the-art stereoscopic image visual comfort enhancement methods. Conclusion The proposed method can automatically implement global linear and local nonlinear disparity remapping processes based on the visual comfort prediction model constructed with different features of stereoscopic images. The proposed method can realize the purpose of improving the visual comfort of stereoscopic images under the premise of ensuring 3D sense, which enhances the overall 3D experience of stereoscopic images.