Depth map inpainting via sparse distortion model

The depth map captured from a real scene by the Kinect motion sensor is always influenced by noise and other environmental factors. As a result, some depth information is missing from the map. This distortion of the depth map directly deteriorates the quality of the virtual viewpoints rendered in 3D video systems. We propose a depth map inpainting algorithm based on a sparse distortion model. First, we train the sparse distortion model using the distortion and real depth maps to obtain two learning dictionaries: one for distortion and one for real depth maps. Second, the sparse coefficients of the distortion and the real depth maps are calculated by orthogonal matching pursuit. We obtain the approximate features of the distortion from the relationship between the learning dictionary and the sparse coefficients of the distortion map. The noisy images are filtered by the joint space structure filter, and the extraction factor is obtained from the resulting image by the extraction factor judgment method. Finally, we combine the learning dictionary and sparse coefficients from the real depth map with the extraction factor to repair the distortion in the depth map. A quality evaluation method is proposed for the original real depth maps with missing pixels. The proposed method achieves better results than comparable methods in terms of depth inpainting and the subjective quality of the rendered virtual viewpoints.     

Decomposing Global Light Transport Using Time of Flight Imaging

Global light transport is composed of direct and indirect components. In this paper, we take the first steps toward analyzing light transport using the high temporal resolution information of time of flight (ToF) images. With pulsed scene illumination, the time profile at each pixel of these images separates different illumination components by their finite travel time and encodes complex interactions between the incident light and the scene geometry with spatially-varying material properties. We exploit the time profile to decompose light transport into its constituent direct, subsurface scattering, and interreflection components. We show that the time profile is well modelled using a Gaussian function for the direct and interreflection components, and a decaying exponential function for the subsurface scattering component. We use our direct, subsurface scattering, and interreflection separation algorithm for five computer vision applications: recovering projective depth maps, identifying subsurface scattering objects, measuring parameters of analytical subsurface scattering models, performing edge detection using ToF images and rendering novel images of the captured scene with adjusted amounts of subsurface scattering.     

基于贝叶斯背景模型的免携带设备目标定位算法

免携带设备目标定位不需要目标携带任何电子设备或标签来对人或其他物体进行定位。针对现有射频层析成像算法在多径环境中定位精度不理想的问题,提出了一种基于贝叶斯背景模型的定位算法。该算法首先将斜拉普拉斯分布和贝叶斯理论相结合来建立贝叶斯背景模型,用于排除冗余链路;然后对接收信号强度的变化值进行加权处理,从而减小多径效应对目标定位的干扰;最后引入目标位置的后验估计均值对目标位置进行修正,提高定位精度。实验结果表明,该定位算法具有可行性和有效性。     

UPF在GPS多路径信号延迟估计中的应用

多径信号延迟估计与多径干扰抑制技术一直是卫星导航领域的一个研究热点。在分析近距离镜面多径信号时间延迟特性的基础上,提出一种基于无迹粒子滤波的多路径信号延迟估计方法。给出标准的粒子滤波算法模型,讨论无迹粒子滤波算法在多路径信号延迟估计中的应用,描述算法的推导过程。计算机模拟结果验证了该算法的有效性。     

基于交通场景区域增强的单幅图像去雾方法

针对当前已有的去雾算法在雾天道路图像的处理上易造成近处路面区域和远处天空区域亮度过低、处理程度偏强,而中远处区域去雾程度较低、亮度过高等问题,以基于深度学习去雾算法为基础提出一种结合雾天道路图像场景深度和道路图像特点的去雾算法。首先基于深度学习的去雾算法原理,构建卷积神经网络求取场景透射率;然后基于大气散射模型和透射率估计出图像深度图,且构造两个参数：上阈值和下阈值来将深度图分为中、远、近三个区域;再基于深度图的不同区域构造增强函数,来确定图像处理的增强幅度照;最后在传统的大气散射模型基础上结合增强幅度照来调节不同区域的复原强度得到优化后的处理图像。实验结果表明,所提算法可以在保证良好去雾效果的前提下增强道路图像的中远处区域,有效解决了去雾后雾天道路图像近处路面和远处天空的色彩失真、对比度过低问题,提升复原图像的视觉效果,并且与暗原色先验算法、均匀与非均匀雾的视觉增强算法以及典型的基于深度学习去雾算法相比具有更好的图像清晰化效果。     

True multi-image alignment and its application to mosaicing andlens distortion correction

Multiple images of a scene are related through 2D/3D view transformations and linear and nonlinear camera transformations. We present an algorithm for true multi-image alignment that does not rely on the measurements of a reference image being distortion free. The algorithm is developed to specifically align and mosaic images using parametric transformations in the presence of lens distortion. When lens distortion is present, none of the images can be assumed to be ideal. In our formulation, all the images are modeled as intensity measurements represented in their respective coordinate systems, each of which is related to an ideal coordinate system through an interior camera transformation and an exterior view transformation. The goal of the accompanying algorithm is to compute an image in the ideal coordinate system while solving for the transformations that relate the ideal system with each of the data images. Key advantages of the technique presented in this paper are: (i) no reliance on one distortion free image, (ii) ability to register images and compute coordinate transformations even when the multiple images are of an extended scene with no overlap between the first and last frame of the sequence, and (iii) ability to handle linear and nonlinear transformations within the same framework. Results of applying the algorithm are presented for the correction of lens distortion, and creation of video mosaics     

复杂阵地的合成导向矢量最大似然测高

雷达在目标低仰角测高时存在严重的多径效应,复杂阵地使多径回波产生无规律反射,造成幅度与相位发生不同程度的畸变.本文引入扰动多径模型,解决经典多径模型与复杂阵地的多径回波反射不匹配问题,研究基于扰动模型的合成导向矢量最大似然(synthesized vector maximum likelihood, SVML)测高方法.该方法引入扰动参数表征复杂阵地的多径回波现象,利用基于稀疏贝叶斯学习的扰动多径(perturbational multipath sparse Bayesian learning,PSBL)算法得到扰动参数,应用于SVML算法,提高了米波雷达在复杂阵地下的测高性能.     

Image-guided ToF depth upsampling: a survey

Recently, there has been remarkable growth of interest in the development and applications of time-of-flight (ToF) depth cameras. Despite the permanent improvement of their characteristics, the practical applicability of ToF cameras is still limited by low resolution and quality of depth measurements. This has motivated many researchers to combine ToF cameras with other sensors in order to enhance and upsample depth images. In this paper, we review the approaches that couple ToF depth images with high-resolution optical images. Other classes of upsampling methods are also briefly discussed. Finally, we provide an overview of performance evaluation tests presented in the related studies.     

Single view metrology of wide-angle lens images

This paper presents a framework and the associated algorithms to perform 3D scene analysis from a single image with lens distortion. Previous work focuses on either making 3D measurements under the assumption of one or more ideal pinhole cameras or correcting the lens distortion up to a projective transformation with no additional metric analyses. In this work, we bridge the gap between these two areas of work by incorporating metric constraints into lens distortion correction to achieve metric calibration. Lens distortion parameters, especially the lens distortion center, can be precisely recovered with this approach. Subsequent 3D measurements can be made from the corrected image to recover scene structures. In addition, we propose an algorithm based on hybrid backward and forward covariance propagation to yield a quantitative analysis of the confidence of the results. Experimental results show that our approach simultaneously performs image correction and 3D scene analysis.     

抗高光的光场深度估计方法

目的 光场相机一次成像可以同时获取场景中光线的空间和角度信息,为深度估计提供了条件。然而,光场图像场景中出现高光现象使得深度估计变得困难。为了提高算法处理高光问题的可靠性,本文提出了一种基于光场图像多视角上下文信息的抗高光深度估计方法。方法 本文利用光场子孔径图像的多视角特性,创建多视角输入支路,获取不同视角下图像的特征信息;利用空洞卷积增大网络感受野,获取更大范围的图像上下文信息,通过同一深度平面未发生高光的区域的深度信息,进而恢复高光区域深度信息。同时,本文设计了一种新型的多尺度特征融合方法,串联多膨胀率空洞卷积特征与多卷积核普通卷积特征,进一步提高了估计结果的精度和平滑度。结果 实验在3个数据集上与最新的4种方法进行了比较。实验结果表明,本文方法整体深度估计性能较好,在4D light field benchmark合成数据集上,相比于性能第2的模型,均方误差（mean square error,MSE）降低了20.24%,坏像素率（bad pixel,BP）降低了2.62%,峰值信噪比（peak signal-to-noise ratio,PSNR）提高了4.96%。同时,通过对CVIA （computer vision and image analysis） Konstanz specular dataset合成数据集和Lytro Illum拍摄的真实场景数据集的定性分析,验证了本文算法的有效性和可靠性。消融实验结果表明多尺度特征融合方法改善了深度估计在高光区域的效果。结论 本文提出的深度估计模型能够有效估计图像深度信息。特别地,高光区域深度信息恢复精度高、物体边缘区域平滑,能够较好地保存图像细节信息。     

室内环境下无线干涉定位系统的多径误差分析

无线干涉定位系统（RIPS）通过获得无线传感网络中节点的相位来实现对节点精确定位。介绍无线干涉定位算法,利用多径效应误差模型实现对RIPS定位结果的多径修正。构建RIPS室内定位仿真模型和硬件平台进行多径环境下的室内测距实验,结果显示实测误差远高于仿真误差。对此,从采样方式、反射系数和节点高度3个方面定量分析造成测距误差的原因。分析结果表明,采样方式、反射系数和定位节点高度设置是造成RIPS实际测量误差的主要因素,经过理论修正后测距误差可以降低87.61%。     

场景线稿动漫效果的自动上色算法

场景线稿具有线条语义多样化的特点,直接应用现有的人像线稿图自动上色算法对其着色容易出现上色错误或棋盘效应等结果失真的现象.针对上述问题,文中提出动漫效果自动上色算法.基于条件生成对抗网络,改进和增强人像线稿图自动上色算法中常用的U型网络(U-Net)生成器的结构,设计双层信息抽取的生成器网络(DIEU-Net),自动完成场景线稿到动漫效果的上色.DIEU-Net设计用于抽取场景线稿浅层显著信息的双卷积子模块(IESS).构建双层IESS与残差结构的集成模块,插入生成器的不同阶段,增强网络在与线稿关联的颜色、位置等重要特征上的全域学习能力,缓和网络加深带来的梯度消失等网络退化问题.同时采用“卷积+上采样”操作替换U-Net生成器中原有的反卷积操作,抑制生成结果中棋盘效应的发生.实验表明,文中算法能较好地克服结果失真的问题,上色效果合理、自然,具有较好的应用推广性,可应用于多种类型景物线稿图的动漫上色.     

K-nearest neighborhood based integration of time-of-flight cameras and passive stereo for high-accuracy depth maps

Both time-of-flight （ToF） cameras and passive stereo can provide the depth information for their corresponding captured real scenes, but they have innate limitations. ToF cameras and passive stereo are intrinsically complementary for certain tasks. It is desirable to appropriately leverage all the available information by ToF cameras and passive stereo. Although some fusion methods have been presented recently, they fail to consider ToF reliability detection and ToF based improvement of passive stereo. As a result, this study proposes an approach to integrating ToF cameras and passive stereo to obtain high-accuracy depth maps. The main contributions are： （1） An energy cost function is devised to use data from ToF cameras to boost the stereo matching of passive stereo; （2） A fusion method is used to combine the depth information from both ToF cameras and passive stereo to obtain high-accuracy depth maps. Experiments show that the proposed approach achieves improved results with high accuracy and robustness.     

Time-of-Flight Cameras in Computer Graphics

A growing number of applications depend on accurate and fast 3D scene analysis. Examples are model and lightfield acquisition, collision prevention, mixed reality and gesture recognition. The estimation of a range map by image analysis or laser scan techniques is still a time-consuming and expensive part of such systems.
A lower-priced, fast and robust alternative for distance measurements are  time-of-flight (ToF)  cameras. Recently, significant advances have been made in producing low-cost and compact ToF devices, which have the potential to revolutionize many fields of research, including computer graphics, computer vision and human machine interaction (HMI).
These technologies are starting to have an impact on research and commercial applications. The upcoming generation of ToF sensors, however, will be even more powerful and will have the potential to become 'ubiquitous real-time geometry devices' for gaming, web-conferencing, and numerous other applications. This paper gives an account of recent developments in ToF technology and discusses the current state of the integration of this technology into various graphics-related applications.     

Detection of sparse targets with structurally perturbed echo dictionaries

In this paper, a novel algorithm is proposed to achieve robust high resolution detection in sparse multipath channels. Currently used sparse reconstruction techniques are not immediately applicable in multipath channel modeling. Performance of standard compressed sensing formulations based on discretization of the multipath channel parameter space degrade significantly when the actual channel parameters deviate from the assumed discrete set of values. To alleviate this off-grid problem, we make use of the particle swarm optimization (PSO) to perturb each grid point that reside in each multipath component cluster. Orthogonal matching pursuit (OMP) is used to reconstruct sparse multipath components in a greedy fashion. Extensive simulation results quantify the performance gain and robustness obtained by the proposed algorithm against the off-grid problem faced in sparse multipath channels.     

Fusion of 2d and 3d sensor data for articulated body tracking

In this article, we present an approach for the fusion of 2d and 3d measurements for model-based person tracking, also known as Human Motion Capture. The applied body model is defined geometrically with generalized cylinders, and is set up hierarchically with connecting joints of different types. The joint model can be parameterized to control the degrees of freedom, adhesion and stiffness. This results in an articulated body model with constrained kinematic degrees of freedom.The fusion approach incorporates this model knowledge together with the measurements, and tracks the target body iteratively with an extended Iterative Closest Point (ICP) approach. Generally, the ICP is based on the concept of correspondences between measurements and model, which is normally exploited to incorporate 3d point cloud measurements. The concept has been generalized to represent and incorporate also 2d image space features.Together with the 3D point cloud from a 3d time-of-flight (ToF) camera, arbitrary features, derived from 2D camera images, are used in the fusion algorithm for tracking of the body. This gives complementary information about the tracked body, enabling not only tracking of depth motions but also turning movements of the human body, which is normally a hard problem for markerless human motion capture systems.The resulting tracking system, named VooDoo is used to track humans in a Human–Robot Interaction (HRI) context. We only rely on sensors on board the robot, i.e. the color camera, the ToF camera and a laser range finder. The system runs in realtime (～20 Hz) and is able to robustly track a human in the vicinity of the robot.     

Increasing the accuracy of Time-of-Flight cameras for machine vision applications

Range imaging based on the Time-of-Flight (ToF) principle evolved largely in recent years. Especially, the lateral resolution, the ability to operate outdoors with sunlight and the sensitivity have been improved. Nevertheless, the acceptance of depth cameras for machine vision in the industry environment is still rather limited. The major shortcoming of ToF depth cameras compared to laser range scanners is their measuring accuracy, which is not sufficient for several applications. In this paper, we firstly introduce several state of the art depth cameras briefly and demonstrate their capabilities. Afterwards, we explore possibilities to increase the radial resolution and the accuracy of ToF depth cameras based on the Photonic Mixer Device (PMD). In general, the usage of higher modulation frequencies promises higher depth resolution but yields on the other hand higher noise levels. Moreover, the accuracy is limited by systematic errors and the measurement are affected by random noise and we show how to minimize and compensate them in industry environments.     

Video Packet Selection and Scheduling for Multipath Streaming

This paper addresses the problem of choosing the best streaming policy for distortion optimal multipath video delivery, under network bandwidth and playback delay constraints. The streaming policy consists in a joint selection of the network path and of the video packets to be transmitted, along with their sending time. A simple streaming model is introduced, which takes into account the video packet importance, and the dependencies between packets. A careful timing analysis allows to compute the quality perceived by the receiver for a constrained playback delay, as a function of the streaming policy. We derive an optimization problem based on a video abstraction model, under the assumption that the server knows, or can predict accurately the state of the network. A detailed analysis of constrained multipath streaming systems provides helpful insights to design an efficient branch and bound algorithm that finds the optimal streaming strategy. This solution allows to bound the performance of any scheduling strategy, but the complexity of the algorithm becomes rapidly intractable. We therefore propose a fast heuristic-based algorithm, built on load-balancing principles. It allows to reach close to optimal performance with a polynomial time complexity. The algorithm is then adapted to live streaming scenarios, where the server has only a partial knowledge of the packet stream, and the channel bandwidth. Extensive simulations show that the proposed algorithm only induces a negligible distortion penalty compared to the optimal strategy, even when the optimization horizon is limited, or the rate estimation is not perfect. Simulation results also demonstrate that the proposed scheduling solution performs better than common scheduling algorithms, and therefore represents a very efficient low-complexity multipath streaming algorithm, for both stored and live video services     

场景切换视频自适应帧间码率控制

针对视频帧中可能出现的大量场景切换,提出一种基于非连接点的场景切换检测算法,提高编码性能,该场景检测算法复杂度低,在运动估计的同时,完成视频场景切换检测。场景切换将导致GOP(group of pictures)长度的变化,并可能出现GOP长度太短的情况。提出改进的自适应GOP时域滤波技术,避免由于GOP太短引起的编码性能下降。针对视频场景切换检测分割出的不同长度的GOP,提出一种基于率失真模型的帧间码率控制算法,利用视频的失真与码率及视频帧复杂度的关系,对帧间码率分配进行优化,提高重构视频帧的总质量。实验结果表明,基于场景检测的自适应帧间码率控制算法能够获得较好的编码性能。     

Optimizing stereo‐to‐multiview conversion for autostereoscopic displays

We present a novel stereo‐to‐multiview video conversion method for glasses‐free multiview displays. Different from previous stereo‐to‐multiview approaches, our mapping algorithm utilizes the limited depth range of autostereoscopic displays optimally and strives to preserve the scene's artistic composition and perceived depth even under strong depth compression. We first present an investigation of how perceived image quality relates to spatial frequency and disparity. The outcome of this study is utilized in a two‐step mapping algorithm, where we (i) compress the scene depth using a non‐linear global function to the depth range of an autostereoscopic display and (ii) enhance the depth gradients of salient objects to restore the perceived depth and salient scene structure. Finally, an adapted image domain warping algorithm is proposed to generate the multiview output, which enables overall disparity range extension.