基于最大后验概率的单视频时间超分辨率重建算法

任何视频摄像设备均具有一定的时间分辨率限制,时间分辨率不足会造成视频中存在运动模糊和运动混叠现象。针对这一问题常用的解决方法是空间去模糊和时间插值,然而这些方法无法从根本上解决问题。提出一种基于最大后验概率(MAP)的单视频时间超分辨率重建方法,该方法通过重建约束来确定条件概率模型,然后利用视频自身具有的时间自相似先验信息得到先验信息模型,最后求得基于最大后验概率的估计值,即通过对单个低时间分辨率视频重建来得到高时间分辨率视频,从而有效解决由于相机曝光时间过长所造成的“运动模糊”和相机帧率不足引起的“运动混叠”现象。通过理论分析与实验,证明了所提方法的有效性。     

视频序列的全景图拼接技术

提出了一种对视频序列进行全景图拼接的方法。主要讨论了有大面积的非刚性运动物体出现的序列，不过此方法也同样适用于无运动物体的纯背景序列。为计算各帧间的投影关系，用仿射模型来描述摄像机运动，并用特征点匹配的方法计算出模型中各参数的值。由于用相关法计算的匹配结果准确率比较低，所以用RANSAC（Random Sampling Consensus）对匹配结果进行了筛选，可以准确求出摄像机运动参数。利用运动参数进行投影，然后用多帧相减并求交集，估计出每帧图像中运动物体存在的区域，最后计算得到了全景图。该方法的结果与前人得到的结果进行了比较，证明用此方法能获得质量较高的全景图。     

Automatic camera control in virtual environments augmented using multiple sparse videos

Automated virtual camera control has been widely used in animation and interactive virtual environments. We have developed a multiple sparse camera based free view video system prototype that allows users to control the position and orientation of a virtual camera, enabling the observation of a real scene in three dimensions (3D) from any desired viewpoint. Automatic camera control can be activated to follow selected objects by the user. Our method combines a simple geometric model of the scene composed of planes (virtual environment), augmented with visual information from the cameras and pre-computed tracking information of moving targets to generate novel perspective corrected 3D views of the virtual camera and moving objects. To achieve real-time rendering performance, view-dependent textured mapped billboards are used to render the moving objects at their correct locations and foreground masks are used to remove the moving objects from the projected video streams. The current prototype runs on a PC with a common graphics card and can generate virtual 2D views from three cameras of resolution 768×576 with several moving objects at about 11 fps.     

Railroad online: acquiring and visualizing route panoramas of rail scenes

A patrol type of surveillance has been performed everywhere from police city patrol to railway inspection. Different from static cameras or sensors distributed in a space, such surveillance has its benefits of low cost, long distance, and efficiency in detecting infrequent changes. However, the challenges are how to archive daily recorded videos in the limited storage space and how to build a visual representation for quick and convenient access to the archived videos. We tackle the problems by acquiring and visualizing route panoramas of rail scenes. We analyze the relation between train motion and the video sampling and the constraints such as resolution, motion blur and stationary blur etc. to obtain a desirable panoramic image. The route panorama generated is a continuous image with complete and non-redundant scene coverage and compact data size, which can be easily streamed over the network for fast access, maneuver, and automatic retrieval in railway environment monitoring. Then, we visualize the railway scene based on the route panorama rendering for interactive navigation, inspection, and scene indexing.     

Outdoor Human Motion Capture by Simultaneous Optimization of Pose and Camera Parameters

We present a method for capturing the skeletal motions of humans using a sparse set of potentially moving cameras in an uncontrolled environment. Our approach is able to track multiple people even in front of cluttered and non‐static backgrounds, and unsynchronized cameras with varying image quality and frame rate. We completely rely on optical information and do not make use of additional sensor information (e.g. depth images or inertial sensors). Our algorithm simultaneously reconstructs the skeletal pose parameters of multiple performers and the motion of each camera. This is facilitated by a new energy functional that captures the alignment of the model and the camera positions with the input videos in an analytic way. The approach can be adopted in many practical applications to replace the complex and expensive motion capture studios with few consumer‐grade cameras even in uncontrolled outdoor scenes. We demonstrate this based on challenging multi‐view video sequences that are captured with unsynchronized and moving (e.g. mobile‐phone or GoPro) cameras.     

Automatic object extraction and reconstruction in active video

A new method of video object extraction is proposed to automatically extract the object of interest from actively acquired videos. Traditional video object extraction techniques often operate under the assumption of homogeneous object motion and extract various parts of the video that are motion consistent as objects. In contrast, the proposed active video object extraction (AVOE) approach assumes that the object of interest is being actively tracked by a non-calibrated camera under general motion and classifies the possible movements of the camera that result in the 2D motion patterns as recovered from the image sequence. Consequently, the AVOE method is able to extract the single object of interest from the active video. We formalize the AVOE process using notions from Gestalt psychology. We define a new Gestalt factor called “shift and hold” and present 2D object extraction algorithms. Moreover, since an active video sequence naturally contains multiple views of the object of interest, we demonstrate that these views can be combined to form a single 3D object regardless of whether the object is static or moving in the video.     

Coded exposure HDR light‐field video recording

Capturing exposure sequences to compute high dynamic range (HDR) images causes motion blur in cases of camera movement. This also applies to light‐field cameras: frames rendered from multiple blurred HDR light‐field perspectives are also blurred. While the recording times of exposure sequences cannot be reduced for a single‐sensor camera, we demonstrate how this can be achieved for a camera array. Thus, we decrease capturing time and reduce motion blur for HDR light‐field video recording. Applying a spatio‐temporal exposure pattern while capturing frames with a camera array reduces the overall recording time and enables the estimation of camera movement within one light‐field video frame. By estimating depth maps and local point spread functions (PSFs) from multiple perspectives with the same exposure, regional motion deblurring can be supported. Missing exposures at various perspectives are then interpolated.     

Semantic Plane-Structure based motion detection with a nonstationary camera

This paper presents a novel method to accurately detect moving objects from a video sequence captured using a nonstationary camera. Although common methods provide effective motion detection for static backgrounds or through only planar-perspective transformation, many detection errors occur when the background contains complex dynamic interferences or the camera undergoes unknown motions. To solve this problem, this study proposed a motion detection method that incorporates temporal motion and spatial structure. In the proposed method, first, spatial semantic planes are segmented, and image registration based on stable background planes is applied to overcome the interferences of the foreground and dynamic background. Thus, the estimated dense temporal motion ensures that small moving objects are not missed. Second, motion pixels are mapped on semantic planes, and then, the spatial distribution constraints of motion pixels, regional shapes and plane semantics, which are integrated into a planar structure, are used to minimise false positives. Finally, based on the dense temporal motion and spatial structure, moving objects are accurately detected. The experimental results on CDnet dataset, Pbi dataset, Aeroscapes dataset, and other challenging self-captured videos under difficult conditions, such as fast camera movement, large zoom variation, video jitters, and dynamic background, revealed that the proposed method can remove background movements, dynamic interferences, and marginal noises and can effectively obtain complete moving objects.© 2017 ElsevierInc.Allrightsreserved.     

视频图像序列运动参数估计与动态拼接

本文采用多重分层叠代算法来估计全局运动参数，并提出应用于动态拼接的运动分割新方法，实现既有摄像机运动又有物体运动的视频图像序列自动拼接。我们的方法基本步骤如下：首先进行全局运动参数的初始估计，并且在分层叠代过程中进行区域分类，得到初始运动模板。接着空间分割原始图像，先根据图像的空间属性由底向上分层合并图像空间区域，再利用视频图像时间属性进一步向上合并，得到图像空间分割结果。然后结合初始运动模板和图像空间分割结果，采用区域分类新方法重新对图像空间分割结果的每个区域进行分类。然后根据分类结果逐步精确求解全局运动参数。最后进行图像合成，得到全景拼接图像。我们的方法利用了多重分层叠代的优点，并且充分考虑到视频图像空间和时间上的属性，实现了运动物体和覆盖背景的精确分割，避免了遮挡问题对全局运动参数估计精度的影响。而且在图像合成时我们解决了拼接图可能产生模糊或某些区域不连续等问题。实验结果表明我们的方法实现了动态视频图像序列高质量的全景拼接。     

Video stabilization using maximally stable extremal region features

Video stabilization is an important technique in present day digital cameras as most of the cameras are hand-held, mounted on moving platforms or subjected to atmospheric vibrations. In this paper we propose a novel video stabilization scheme based on estimating the camera motion using maximally stable extremal region features. These features traditionally used in wide baseline stereo problems were never explored for video stabilization purposes. Through our extensive experiments show we how some properties of these region features are suitable for the stabilization task. After estimating the global camera motion parameters using these region features, we smooth the motion parameters using a gaussian filter to retain the desired motion. Finally, motion compensation is carried out to obtain a stabilized video sequence. A number of examples on real and synthetic videos demonstrate the effectiveness of our proposed approach. We compare our results to existing techniques and show how our proposed approach compares favorably to them. Interframe Transformation Fidelity is used for objective evaluation of our proposed approach.     

Controlling Motion Blur in Synthetic Long Time Exposures

In a photo, motion blur can be used as an artistic style to convey motion and to direct attention. In panning or tracking shots, a moving object of interest is followed by the camera during a relatively long exposure. The goal is to get a blurred background while keeping the object sharp. Unfortunately, it can be difficult to impossible to precisely follow the object. Often, many attempts or specialized physical setups are needed. This paper presents a novel approach to create such images. For capturing, the user is only required to take a casually recorded hand‐held video that roughly follows the object. Our algorithm then produces a single image which simulates a stabilized long time exposure. This is achieved by first warping all frames such that the object of interest is aligned to a reference frame. Then, optical flow based frame interpolation is used to reduce ghosting artifacts from temporal undersampling. Finally, the frames are averaged to create the result. As our method avoids segmentation and requires little to no user interaction, even challenging sequences can be processed successfully. In addition, artistic control is available in a number of ways. The effect can also be applied to create videos with an exaggerated motion blur. Results are compared with previous methods and ground truth simulations. The effectiveness of our method is demonstrated by applying it to hundreds of datasets. The most interesting results are shown in the paper and in the supplemental material.     

Intended human object detection for automatically protecting privacy in mobile video surveillance

With the recent popularization of mobile video cameras including camera phones, a new technology, mobile video surveillance, which uses mobile video cameras for video surveillance has been emerging. Such videos, however, may infringe upon the privacy of others by disclosing privacy sensitive information (PSI), i.e., their appearances. To prevent videos from infringing on the right to privacy, new techniques are required that automatically obscure PSI regions. The problem is how to determine the PSI regions to be obscured while maintaining enough video content to present the camera persons’ capture-intentions, i.e., what they want to record in their videos to achieve their surveillance tasks. To this end, we introduce a new concept called intended human objects that are defined as human objects essential for capture-intentions, and develop a new method called intended human object detection that automatically detects the intended human objects in videos taken by different camera persons. Through the process of intended human object detection, we develop a system for automatically obscuring PSI regions. We experimentally show the performance of intended human object detection and the contributions of the features used. Our user study shows the potential applicability of our proposed system.     

运动目标三维轨迹可视化与关联分析方法

随着治安监控系统的普及,越来越多的监控摄像头被安装在各个交通道路和公共场所中,每天都产生大量的监控视频.如今,监控视频分析工作主要是采用人工观看的方式来排查异常,以这种方式来分析视频内容耗费大量的人力和时间.目前,关于视频分析方面的研究大多是针对目标个体的异常行为检测和追踪,缺乏针对对象之间的关联关系的分析,对视频中的一些对象和场景之间的关联关系等还没有较为有效的表示和分析方法.针对这一现状,提出一种基于运动目标三维轨迹的关联视频可视分析方法来辅助人工分析视频,首先对视频资料进行预处理,获取各个目标对象的运动轨迹信息,由于二维轨迹难以处理轨迹的自相交、循环运动和停留等现象,并且没有时间信息就难以对同一空间内多个对象轨迹进行的关联性分析,于是结合时间维度对轨迹进行三维化扩展.该方法支持草图交互方式来操作,在分析过程中进行添加草图注释来辅助分析.可结合场景和对象的时空关系对轨迹进行关联性计算,得出对象及场景之间的关联模型,通过对对象在各个场景出现状况的统计,结合人工预先设定的规则,可实现对异常行为报警,辅助用户决策.     

An edge detection framework conjoining with IMU data for assisting indoor navigation of visually impaired persons

Smartphone applications based on object detection techniques have recently been proposed to assist visually impaired persons with navigating indoor environments. In the smartphone, digital cameras are installed to detect objects which are important for navigation. Prior to detect the interested objects from images, edges on the objects have to be identified. Object edges are difficult to be detected accurately as the image is contaminated by strong image blur which is caused by camera movement. Although deblurring algorithms can be used to filter blur noise, they are computationally expensive and not suitable for real-time implementation. Also edge detection algorithms are mostly developed for stationary images without serious blur. In this paper, a modified sigmoid function (MSF) framework based on inertial measurement unit (IMU) is proposed to mitigate these problems. The IMU estimates blur levels to adapt the MSF which is computationally simple. When the camera is moving, the topological structure of the MSF is estimated continuously in order to improve effectiveness of edge detections. The performance of the MSF framework is evaluated by detecting object edges on video sequences associated with IMU data. The MSF framework is benchmarked against existing edge detection techniques and results show that it can obtain comparably lower errors. It is further shown that the computation time is significantly decreased compared to using techniques that deploy deblurring algorithms, thus making our proposed technique a strong candidate for reliable real-time navigation.     

Exploring Defocus Matting: Nonparametric Acceleration, Super-Resolution, and Off-Center Matting

Defocus matting is a fully automatic and passive method for pulling mattes from video captured with coaxial cameras that have different depths of field and planes of focus. Nonparametric sampling can accelerate the video-matting process from minutes to seconds per frame. In addition, a super-resolution technique efficiently bridges the gap between mattes from high-resolution video cameras and those from low-resolution cameras. Off-center matting pulls mattes for an external high-resolution camera that doesn't share the same center of projection as the low-resolution cameras used to capture the defocus matting data. In this article, we address these limitations and extend defocus matting in several important ways     

Learning Layered Motion Segmentations of Video

We present an unsupervised approach for learning a layered representation of a scene from a video for motion segmentation. Our method is applicable to any video containing piecewise parametric motion. The learnt model is a composition of layers, which consist of one or more segments. The shape of each segment is represented using a binary matte and its appearance is given by the rgb value for each point belonging to the matte. Included in the model are the effects of image projection, lighting, and motion blur. Furthermore, spatial continuity is explicitly modeled resulting in contiguous segments. Unlike previous approaches, our method does not use reference frame(s) for initialization. The two main contributions of our method are: (i) A novel algorithm for obtaining the initial estimate of the model by dividing the scene into rigidly moving components using efficient loopy belief propagation; and (ii) Refining the initial estimate using α β-swap and α-expansion algorithms, which guarantee a strong local minima. Results are presented on several classes of objects with different types of camera motion, e.g. videos of a human walking shot with static or translating cameras. We compare our method with the state of the art and demonstrate significant improvements.     

DeepDSAIR: Deep 6-DOF camera relocalization using deblurred semantic-aware image representation for large-scale outdoor environments

Deep Learning methods can deploy a fast, robust and lightweight model to solve the problem of 6-DOF camera relocalization in large-scale outdoor environments. However, two significant characteristics of captured images in a large-scale outdoor environment are moving objects, which should not include in the representation of an environment, and also motion blur which widely exists in the images captured with moving cameras. None of the existing approaches study and investigate these two problems in their method. This paper, for the first time, proposes a deep network architecture that is trained based on the knowledge achieved by combining deblurring and semantic segmentation modules and examines the effect of this combination on a challenging dataset. Results show approximately 20 and 50% improvement in camera position and orientation re-localization error respectively.     

Characterizing displays by their temporal aperture: A theoretical framework

Abstract— The spatio‐temporal aperture and sample rate of a video display determines both the static and dynamic resolution of the video signal that is rendered. The dynamic display characteristics like the visibility of large‐area flicker, motion judder, and motion blur can be derived from the frame rate and the temporal extent of the pixel aperture (i.e., the temporal aperture). For example, liquid‐crystal displays (LCDs) have an aperture that is relatively small in the spatial dimension and wide in the temporal domain. Consequently, moving objects displayed on an LCD suffer from motion blur. Especially in TV applications, the temporal dimension has a large impact on the overall picture quality. The temporal aperture, together with the frame rate, is shown to predict the amount of perceived large‐area flicker, motion judder, and motion blur and also the performance of motion‐blur reduction algorithms for LCDs. From this analysis it is further determined how to obtain the optimal temporal aperture of a television display, for which not only properties of the human visual system (HVS), but also the properties of the video signal have to be taken into account.     

一种水下运动物体三维轨迹视觉测量方法

针对海洋工程中采用的设备深海悬垂法安装过程,采用多摄像头视频运动分析方 法计算水下三维运动轨迹可用于指导海洋工程的结构安装和分析设备水下运动特征。水下视频 和图像的处理获取面临着诸多挑战,首先由于水下环境悬浮物和颗粒较多,光在水下发生了散 射,使水下图像发生了退化;其次水下视频运动分析遇到的一个主要障碍是光线的折射引起的 图像误差。由于光在水、玻璃、空气不同介质间发生折射,光路发生弯曲,陆地上的摄像机成 像模型在水中不再适用,需要提出新的水下摄像机成像模型。本文引入带光线折射的水下摄像 机成像模型,研究水下摄像机的内参数和外参数标定方法,利用固定布置的 3 个水下摄像机拍 摄的目标水下运动视频来计算水下目标的轨迹。该方法适用于水池环境下水下物体大范围运动, 可以得到较为精确的轨迹,并得到了实验验证。     

Systems and algorithms for autonomous and scalable crowd surveillance using robotic PTZ cameras assisted by a wide-angle camera

We report an autonomous surveillance system with multiple pan-tilt-zoom (PTZ) cameras assisted by a fixed wide-angle camera. The wide-angle camera provides large but low resolution coverage and detects and tracks all moving objects in the scene. Based on the output of the wide-angle camera, the system generates spatiotemporal observation requests for each moving object, which are candidates for close-up views using PTZ cameras. Due to the fact that there are usually much more objects than the number of PTZ cameras, the system first assigns a subset of the requests/objects to each PTZ camera. The PTZ cameras then select the parameter settings that best satisfy the assigned competing requests to provide high resolution views of the moving objects. We propose an approximation algorithm to solve the request assignment and the camera parameter selection problems in real time. The effectiveness of the proposed system is validated in both simulation and physical experiment. In comparison with an existing work using simulation, it shows that in heavy traffic scenarios, our algorithm increases the number of observed objects by over 210%.