On the Apparent Transparency of a Motion Blurred Object

An object which moves during the exposure time results in a blurred smear in the image. We consider the smear as if it was the image of a semitransparent object, and we retrieve its alpha matte by means of known techniques. The alpha value at a pixel is meaningfully interpreted as the fraction of the exposure time during which the object projection overlapped that pixel. Basing on this fact, our work highlights interesting qualitative and quantitative properties of the alpha matte, which can be used to derive constraints on the object’s apparent contour, and its motion during the exposure time, from a single motion-blurred image; we also show that some of these properties hold on the original image. The theory is validated with experimental results both on synthetic and real images, highlighting strengths and limitations; we point out a range of possible applications, including blurred image interpretation, temporal superresolution of object contours, model-based reconstruction of nontrivial motion, and improvements of alpha matting techniques.     

Flexible and Accurate Transparent‐Object Matting and Compositing Using Refractive Vector Field

In digital image editing, environment matting and compositing are fundamental and interesting operations that can capture and simulate the refraction and reflection effects of light from an environment. The state‐of‐the‐art real‐time environment matting and compositing method is short of flexibility, in the sense that it has to repeat the entire complex matte acquisition process if the distance between the object and the background is different from that in the acquisition stage, and also lacks accuracy, in the sense that it can only remove noises but not errors. In this paper, we introduce the concept of refractive vector and propose to use a refractive vector field as a new representation for environment matte. Such refractive vector field provides great flexibility for transparent‐object environment matting and compositing. Particularly, with only one process of the matte acquisition and the refractive vector field extraction, we are able to composite the transparent object into an arbitrary background at any distance. Furthermore, we introduce a piecewise vector field fitting algorithm to simultaneously remove both noises and errors contained in the extracted matte data. Experimental results show that our method is less sensitive to artefacts and can generate perceptually good composition results for more general scenarios.     

Creating Fluid Animation from a Single Image using Video Database

We present a method for synthesizing fluid animation from a single image, using a fluid video database. The user inputs a target painting or photograph of a fluid scene along with its alpha matte that extracts the fluid region of interest in the scene. Our approach allows the user to generate a fluid animation from the input image and to enter a few additional commands about fluid orientation or speed. Employing the database of fluid examples, the core algorithm in our method then automatically assigns fluid videos for each part of the target image. Our method can therefore deal with various paintings and photographs of a river, waterfall, fire, and smoke. The resulting animations demonstrate that our method is more powerful and efficient than our prior work.     

Local Shape from Mirror Reflections

We study the problem of recovering the 3D shape of an unknown smooth specular surface from a single image. The surface reflects a calibrated pattern onto the image plane of a calibrated camera. The pattern is such that points are available in the image where position, orientations, and local scale may be measured (e.g. checkerboard). We first explore the differential relationship between the local geometry of the surface around the point of reflection and the local geometry in the image.We then study the inverse problem and give necessary and sufficient conditions for recovering surface position and shape.We prove that surface position and shape up to third order can be derived as a function of local position, orientation and local scale measurements in the image when two orientations are available at the same point (e.g. a corner). Information equivalent to scale and orientation measurements can be also extracted from the reflection of a planar scene patch of arbitrary geometry, provided that the reflections of (at least) 3 distinctive points may be identified.We validate our theoretical results with both numerical simulations and experiments with real surfaces.     

A closed-form solution to natural image matting

Interactive digital matting, the process of extracting a foreground object from an image based on limited user input, is an important task in image and video editing. From a computer vision perspective, this task is extremely challenging because it is massively ill-posed -- at each pixel we must estimate the foreground and the background colors, as well as the foreground opacity ("alpha matte") from a single color measurement. Current approaches either restrict the estimation to a small part of the image, estimating foreground and background colors based on nearby pixels where they are known, or perform iterative nonlinear estimation by alternating foreground and background color estimation with alpha estimation.In this paper we present a closed-form solution to natural image matting. We derive a cost function from local smoothness assumptions on foreground and background colors, and show that in the resulting expression it is possible to analytically eliminate the foreground and background colors to obtain a quadratic cost function in alpha. This allows us to find the globally optimal alpha matte by solving a sparse linear system of equations. Furthermore, the closed-form formula allows us to predict the properties of the solution by analyzing the eigenvectors of a sparse matrix, closely related to matrices used in spectral image segmentation algorithms. We show that high quality mattes for natural images may be obtained from a small amount of user input.     

Preserving global features of fluid animation from a single image using video examples

We synthesize animations from a single image by transferring fluid motion of a video example globally.Given a target image of a fluid scene,an alpha matte is required to extract the fluid region.Our method needs to adjust a user-specified video example for producing the fluid motion suitable for the extracted fluid region.Employing the fluid video database,the flow field of the target image is obtained by warping the optical flow of a video frame that has a visually similar scene to the target image according to their scene correspondences,which assigns fluid orientation and speed automatically.Results show that our method is successful in preserving large fluid features in the synthesized animations.In comparison to existing approaches,it is both possible and useful to utilize our method to create flow animations with higher quality.     

Content-aware copying and pasting in images

We present a content-aware image copy-and-paste technique which combines ideas from both matting and gradient-based methods. We modify the diffusion process used in the gradient-based approach to use the alpha matte for the cloned area as a weight function to control intensity interpolation. This ensures that the color style of the significant parts of the selected region is preserved when pasting. We use a framework based on mean-value coordinates to implement our approach, allowing us to provide a parallel implementation suitable for use on a GPU. Experimental results show the advantages of our method.     

双边滤波的改进高光去除

目的 由于非均匀光照条件下,物体表面通常出现块状的强反射区域,传统的去高光方法在还原图像时容易造成颜色失真或者边缘的丢失。针对这些缺点,提出一种改进的基于双边滤波的去高光方法。方法 首先通过双色反射模型变换得到镜面反射分量与最大漫反射色度之间的转换关系,然后利用阈值将图像的像素点分为两类,将仅含漫反射分量的像素点与含有镜面反射分量的像素点分离开来,对两类像素点的最大漫反射色度分别做估计,接着以估计的最大漫反射色度的相似度作为双边滤波器的值域,同时以图像的最大色度图作为双边滤波的引导图保边去噪,进而达到去除镜面反射分量的目的。结果 以经典的高光图像作为处理对象,对含有镜面反射和仅含漫反射的像素点分别做最大漫反射色度估计,再以该估计图作为双边滤波的引导图,不仅能去除镜面反射分量还能有效的保留图像的边缘信息,最大程度的还原图像细节颜色,并且解决了原始算法处理结果中R、G、B三通道相似的像素点所出现的颜色退化问题。用改进的双边滤波去高光算法对50幅含高光的图像做处理,并将该算法与Yang方法和Shen方法分别作对比,结果图的峰值信噪比（PSNR）也分别平均提高4.17%和8.40%,所提算法的处理效果更符合人眼视觉,图像质量更好。结论 实验结果表明针对含镜面反射的图像,本文方法能够更有效去除图像的多区域局部高光,完成对图像的复原,可为室内外光照不匀情况下所采集图像的复原提供有效理论基础。     

存在镜面反射时的立体匹配研究

传统的立体匹配方法建立在Lambertian的漫反射模型之上,漫反射模型的立体匹配在一个图像中大部分是有效的,但是在处理图像中包含镜面反射部分时结果会产生严重的匹配错误.为了解决个问题,根据二色反射模型引入一种漫反射和镜面反射的分离方法,匹配图像中存在镜面反射部分时先滤除掉镜面反射再进行匹配,在镜面反射部分也能匹配得到正确的视差.实验结果证明该方法很有效.     

Motion regularization for matting motion blurred objects

This paper addresses the problem of matting motion blurred objects from a single image. Existing single image matting methods are designed to extract static objects that have fractional pixel occupancy. This arises because the physical scene object has a finer resolution than the discrete image pixel and therefore only occupies a fraction of the pixel. For a motion blurred object, however, fractional pixel occupancy is attributed to the object’s motion over the exposure period. While conventional matting techniques can be used to matte motion blurred objects, they are not formulated in a manner that considers the object’s motion and tend to work only when the object is on a homogeneous background. We show how to obtain better alpha mattes by introducing a regularization term in the matting formulation to account for the object’s motion. In addition, we outline a method for estimating local object motion based on local gradient statistics from the original image. For the sake of completeness, we also discuss how user markup can be used to denote the local direction in lieu of motion estimation. Improvements to alpha mattes computed with our regularization are demonstrated on a variety of examples.     

Shared Sampling for Real‐Time Alpha Matting

Image matting aims at extracting foreground elements from an image by means of color and opacity (alpha) estimation. While a lot of progress has been made in recent years on improving the accuracy of matting techniques, one common problem persisted: the low speed of matte computation. We present the first real‐time matting technique for natural images and videos. Our technique is based on the observation that, for small neighborhoods, pixels tend to share similar attributes. Therefore, independently treating each pixel in the unknown regions of a trimap results in a lot of redundant work. We show how this computation can be significantly and safely reduced by means of a careful selection of pairs of background and foreground samples. Our technique achieves speedups of up to two orders of magnitude compared to previous ones, while producing high‐quality alpha mattes. The quality of our results has been verified through an independent benchmark. The speed of our technique enables, for the first time, real‐time alpha matting of videos, and has the potential to enable a new class of exciting applications.     

Efficient Shadow Removal Using Subregion Matching Illumination Transfer

This paper proposes a new shadow removal approach for input single natural image by using subregion matching illumination transfer We first propose an effective and automatic shadow detection algorithm incorporating global successive thresholding scheme and local boundary refinement. Then we present a novel shadow removal algorithm by performing illumination transfer on the matched subregion pairs between the shadow regions and non‐shadow regions, and this method can process complex images with different kinds of shadowed texture regions and illumination conditions. In addition, we develop an efficient shadow boundary processing method by using alpha matte interpolation, which produces seamless transition between the shadow and non‐shadow regions. Experimental results demonstrate the capabilities of our algorithm in both the shadow removal quality and performance.     

Chromaticity-based separation of reflection components in a single image

The separation of diffuse and specular reflection components, or equivalently specularity removal, is required in the fields of computer vision, object recognition and image synthesis. This paper proposes a simple and effective method to separate reflections in a color image based on the error analysis of chromaticity and appropriate selection of body color for each pixel. By solving the least-squares problem of the dichromatic reflection model, reflection separation is implemented on a single pixel level, without requiring image segmentation and even local interactions between neighboring pixels. Experimental evaluation indicates that the proposed method is effective and can deal with a wide variety of images.     

改进融合策略下透明度引导的逆光图像增强

目的 针对传统的逆光图像增强算法存在的曝光正常区域与逆光区域间阈值计算复杂、分割精度不足、过度曝光以及增强不足等问题,提出一种改进融合策略下透明度引导的逆光图像增强算法。方法 对逆光图像在HSV(hue, saturation, value)空间中的亮度分量进行亮度提升和对比度增强,然后通过金字塔融合策略对改进的亮度分量进行分解和重构,恢复逆光区域的细节和颜色信息。此外,利用深度抠图网络计算透明度蒙版,对增强的逆光区域与源图像进行融合处理,维持非逆光区域亮度不变。通过改进融合策略增强的图像在透明度引导下既有效恢复了逆光区域又避免了曝光过度的问题。结果 实验在多幅逆光图像上与直方图均衡算法、MSR (multi-scale Retinex)、Zero-DEC (zero-reference deep curve estimation)、AGLLNet (attention guided low-light image enhancement)和LBR (learning-based restoration) 5种方法进行了比较,在信息熵(information entropy,IE)和盲...     

Segmentation of Colour Images with Highlights and Shadows sing Fuzzy-like Reasoning

In this paper, a fuzzy-like technique is presented that resolves several difficult issues related to image segmentation, such as highlights and shadows. Large, relatively continuous, areas within an image are usually easy to segment, and the pixels included within different segments are often determined by using derived edge information. However, in many cases, pixels which lie between segments or in high frequency areas of an image cannot be easily categorised as belonging to any particular segments. Typically, according to the dichromatic reflection model, these pixels may belong to the matte, highlight or shadow area of the closest segment; or, in association with neighbouring pixels, they make up a separate smaller segment. The dichromatic reflection model is applied here to merge highlight and shadow areas with matte areas in an image. By segmenting those pixels into proper regions, the proposed fuzzy-like reasoning approach provides a more human-like segmentation of images.     

A Physically‐Based BSDF for Modeling the Appearance of Paper

We present a novel appearance model for paper. Based on our appearance measurements for matte and glossy paper, we find that paper exhibits a combination of subsurface scattering, specular reflection, retroreflection, and surface sheen. Classic microfacet and simple diffuse reflection models cannot simulate the double‐sided appearance of a thin layer. Our novel BSDF model matches our measurements for paper and accounts for both reflection and transmission properties. At the core of the BSDF model is a method for converting a multi‐layer subsurface scattering model (BSSRDF) into a BSDF, which allows us to retain physically‐based absorption and scattering parameters obtained from the measurements. We also introduce a method for computing the amount of light available for subsurface scattering due to transmission through a rough dielectric surface. Our final model accounts for multiple scattering, single scattering, and surface reflection and is capable of rendering paper with varying levels of roughness and glossiness on both sides.     

Separating reflection components based on chromaticity and noise analysis

Many algorithms in computer vision assume diffuse only reflections and deem specular reflections to be outliers. However, in the real world, the presence of specular reflections is inevitable since there are many dielectric inhomogeneous objects which have both diffuse and specular reflections. To resolve this problem, we present a method to separate the two reflection components. The method is principally based on the distribution of specular and diffuse points in a two-dimensional maximum chromaticity-intensity space. We found that, by utilizing the space and known illumination color, the problem of reflection component separation can be simplified into the problem of identifying diffuse maximum chromaticity. To be able to identity the diffuse maximum chromaticity correctly, an analysis of the noise is required since most real images suffer from it. Unlike existing methods, the proposed method can separate the reflection components robustly for any kind of surface roughness and light direction.     

Detection of diffuse and specular interface reflections and inter-reflections by color image segmentation

We present a computational model and algorithm for detecting diffuse and specular interface reflections and some inter-reflections. Our color reflection model is based on the dichromatic model for dielectric materials and on a color space, called S space, formed with three orthogonal basis functions. We transform color pixels measured in RGB into the S space and analyze color variations on objects in terms of brightness, hue and saturation which are defined in the S space. When transforming the original RGB data into the S space, we discount the scene illumination color that is estimated using a white reference plate as an active probe. As a result, the color image appears as if the scene illumination is white. Under the whitened illumination, the interface reflection clusters in the S space are all aligned with the brightness direction. The brightness, hue and saturation values exhibit a more direct correspondence to body colors and to diffuse and specular interface reflections, shading, shadows and inter-reflections than the RGB coordinates. We exploit these relationships to segment the color image, and to separate specular and diffuse interface reflections and some inter-reflections from body reflections. The proposed algorithm is effications for uniformly colored dielectric surfaces under singly colored scene illumination. Experimental results conform to our model and algorithm within the liminations discussed.     

改进的自然图像鲁棒抠图算法

自然抠图从背景复杂的彩色图像中根据已知像素进行未知像素的透明度估计以实现前景的准确提取,是图像处理和影视制作的关键技术之一。由于自然图像透明度值的求解具有高度不确定性,目前对颜色和结构复杂的图像抠图效果并不理想。提出一种改进的鲁棒抠图算法。该方法根据trimap的已知前景和背景区域减少未知像素个数;计算前景-背景对的可靠性,选择可信赖的样本对获得透明度的初始值和信心值;优化基于图拉普拉斯的二次目标函数并确定未知像素最终透明度值。自然抠图实验结果表明,与几种主要抠图算法相比,所提方法能提取到具有更好视觉效果和均方误差的前景。     

The measurement of highlights in color images

In this paper, we present an approach to color image understanding that accounts for color variations due to highlights and shading. We demonstrate that the reflected light from every point on a dielectric object, such as plastic, can be described as a linear combination of the object color and the highlight color. The colors of all light rays reflected from one object then form a planar cluster in the color space. The shape of this cluster is determined by the object and highlight colors and by the object shape and illumination geometry. We present a method that exploits the difference between object color and highlight color to separate the color of every pixel into a matte component and a highlight component. This generates two intrinsic images, one showing the scene without highlights, and the other one showing only the highlights. The intrinsic images may be a useful tool for a variety of algorithms in computer vision, such as stereo vision, motion analysis, shape from shading, and shape from highlights. Our method combines the analysis of matte and highlight reflection with a sensor model that accounts for camera limitations. This enables us to successfully run our algorithm on real images taken in a laboratory setting. We show and discuss the results.This material is based upon work supported by the National Science Foundation under Grant DCR-8419990 and by the Defense Advanced Research Projects Agency (DOD), ARPA Order No. 4976, monitored by the Air Force Avionics Laboratory under contract F33615-84-K-1520. Any opinions, findings, and conclusions or recommendations expressed in this publication are those of the authors and do not necessarily reflect the views of the National Science Foundation, the Defense Advanced Research Projects Agency, or the US Government.