Intrinsic Images by Clustering

Decomposing an input image into its intrinsic shading and reflectance components is a long‐standing ill‐posed problem. We present a novel algorithm that requires no user strokes and works on a single image. Based on simple assumptions about its reflectance and luminance, we first find clusters of similar reflectance in the image, and build a linear system describing the connections and relations between them. Our assumptions are less restrictive than widely‐adopted Retinex‐based approaches, and can be further relaxed in conflicting situations. The resulting system is robust even in the presence of areas where our assumptions do not hold. We show a wide variety of results, including natural images, objects from the MIT dataset and texture images, along with several applications, proving the versatility of our method.     

GEMSe: Visualization‐Guided Exploration of Multi‐channel Segmentation Algorithms

We present GEMSe, an interactive tool for exploring and analyzing the parameter space of multi‐channel segmentation algorithms. Our targeted user group are domain experts who are not necessarily segmentation specialists. GEMSe allows the exploration of the space of possible parameter combinations for a segmentation framework and its ensemble of results. Users start with sampling the parameter space and computing the corresponding segmentations. A hierarchically clustered image tree provides an overview of variations in the resulting space of label images. Details are provided through exemplary images from the selected cluster and histograms visualizing the parameters and the derived output in the selected cluster. The correlation between parameters and derived output as well as the effect of parameter changes can be explored through interactive filtering and scatter plots. We evaluate the usefulness of GEMSe through expert reviews and case studies based on three different kinds of datasets: A synthetic dataset emulating the combination of 3D X‐ray computed tomography with data from K‐Edge spectroscopy, a three‐channel scan of a rock crystal acquired by a Talbot‐Lau grating interferometer X‐ray computed tomography device, as well as a hyperspectral image.     

Retrieval and Visualization of Human Motion Data via Stick Figures

We propose 2D stick figures as a unified medium for visualizing and searching for human motion data. The stick figures can express a wide range or human motion, and they are easy to be drawn by people without any professional training. In our interface, the user can browse overall motion by viewing the stick figure images generated from the database and retrieve them directly by using sketched stick figures as an input query. We started with a preliminary survey to observe how people draw stick figures. Based on the rules observed from the user study, we developed an algorithm converting motion data to a sequence of stick figures. The feature‐based comparison method between the stick figures provides an interactive and progressive search for the users. They assist the user's sketching by showing the current retrieval result at each stroke. We demonstrate the utility of the system with a user study, in which the participants retrieved example motion segments from the database with 102 motion files by using our interface.     

SecondSkin: An interactive method for appearance transfer

SecondSkin estimates an appearance model for an object visible in a video sequence, without the need for complex interaction or any calibration apparatus. This model can then be transferred to other objects, allowing a non‐expert user to insert a synthetic object into a real video sequence so that its appearance matches that of an existing object, and changes appropriately throughout the sequence. As the method does not require any prior knowledge about the scene, the lighting conditions, or the camera, it is applicable to video which was not captured with this purpose in mind. However, this lack of prior knowledge precludes the recovery of separate lighting and surface reflectance information. The SecondSkin appearance model therefore combines these factors. The appearance model does require a dominant light‐source direction, which we estimate via a novel process involving a small amount of user interaction. The resulting model estimate provides exactly the information required to transfer the appearance of the original object to new geometry composited into the same video sequence.     

A Dynamic Noise Primitive for Coherent Stylization

We present a new solution for temporal coherence in non‐photorealistic rendering (NPR) of animations. Given the conflicting goals of preserving the 2D aspect of the style and the 3D scene motion, any such solution is a tradeoff. We observe that primitive‐based methods in NPR can be seen as texture‐based methods when using large numbers of primitives, leading to our key insight, namely that this process is similar to sparse convolution noise in procedural texturing. Consequently, we present a new primitive for NPR based on Gabor noise, that preserves the 2D aspect of noise, conveys the 3D motion of the scene, and is temporally continuous. We can thus use standard techniques from procedural texturing to create various styles, which we show for interactive NPR applications. We also present a user study to evaluate this and existing solutions, and to provide more insight in the trade‐off implied by temporal coherence. The results of the study indicate that maintaining coherent motion is important, but also that our new solution provides a good compromise between the 2D aspect of the style and 3D motion.     

Interactive Physics‐based Ink Splattering Art Creation

This paper presents an interactive system for ink splattering, a form of abstract art that artists splat ink onto the canvas. The default input device of our system is a pressure‐sensitive 2D stylus, the most common sketching tool for digital artists, and we propose two interaction mode: ink‐flicking mode and ink‐dripping mode , that are designed to be analogous to the artistic techniques of ink splattering in real world. The core of our ink splattering system is a novel three‐stage ink splattering framework that simulates the physics‐based interaction of ink with different mediums including brush heads, air and paper. We have implemented the physical engine in CUDA and the whole simulation process runs at interactive speed.     

Interactive Cover Design Considering Physical Constraints

We developed an interactive system to design a customized cover for a given three‐dimensional (3D) object such as a camera, teapot, or car. The system first computes the convex hull of the input geometry. The user segments it into several cloth patches by drawing on the 3D surface. This paper provides two technical contributions. First, it introduces a specialized flattening algorithm for cover patches. It makes each two‐dimensional edge in the flattened pattern equal to or longer than the original 3D edge; a smaller patch would fail to cover the object, and a larger patch would result in extra wrinkles. Second, it introduces a mechanism to verify that the user‐specified opening would be large enough for the object to be removed. Starting with the initial configuration, the system virtually “pulls” the object out of the cover while avoiding excessive stretching of cloth patches. We used the system to design real covers and confirmed that it functions as intended.     

Rephotography Using Image Collections

This paper proposes a novel system that “rephotographs” a historical photograph with a collection of images. Rather than finding the accurate viewpoint of the historical photo, users only need to take a number of photographs around the target scene. We adopt the structure from motion technique to estimate the spatial relationship among these photographs, and construct a set of 3D point cloud. Based on the user‐specified correspondences between the projected 3D point cloud and historical photograph, the camera parameters of the historical photograph are estimated. We then combine forward and backward warping images to render the result. Finally, inpainting and content‐preserving warping are used to refine it, and the photograph at the same viewpoint of the historical one is produced by this photo collection.     

Digital Camouflage Images Using Two‐scale Decomposition

We present an alternative approach to create digital camouflage images which follows human's perception intuition and complies with the physical creation procedure of artists. Our method is based on a two‐scale decomposition scheme of the input images. We modify the large‐scale layer of the background image by considering structural importance based on energy optimization and the detail layer by controlling its spatial variation. A gradient correction is presented to prevent halo artifacts. Users can control the difficulty level of perceiving the camouflage effect through a few parameters. Our camouflage images are natural and have less long coherent edges in the hidden region. Experimental results show that our algorithm yields visually pleasing camouflage images.     

Still‐Frame Simulation for Fire Effects of Images

We propose various simulation strategies to generate single‐frame fire effects for images, as opposed to multi‐frame fire effects for animations. To accelerate 3D simulation and to provide a user with early hints on the final effect, we propose a 2D‐guided 3D simulation approach, which runs a faster 2D simulation first, and then guides 3D simulation using the 2D simulation result. To achieve this, we explore various boundary conditions and develop a constrained projection method. Since only the final frame will be used while intermediate frames are abandoned, earlier intermediate frames can take larger time steps and have large noise applied, quickly generating turbulent flow structures. As the final frame approaches, we increase the flow quality by reducing the time step and not adding any noise. This adaptive time stepping allows us to use more computational resource near or at the final frame. We also develop divergence and buoyancy modification methods to guide flames along arbitrary, even physically implausible, directions. Our simulation methods can effectively and efficiently generate a variety of fire effects useful for image decoration.     

Generating Pointillism Paintings Based on Seurat's Color Composition

This paper presents a novel example‐based stippling technique that employs a simple and intuitive concept to convert a color image into a pointillism painting. Our method relies on analyzing and imitating the color distributions of Seurat's paintings to obtain a statistical color model. Then, this model can be easily combined with the modified multi‐class blue noise sampling to stylize an input image with characteristics of color composition in Seurat's paintings. The blue noise property of the output image also ensures that the color points are randomly located but remain spatially uniform. In our experiments, the multivariate goodness‐of‐fit tests were adopted to quantitatively analyze the results of the proposed and previous methods, further confirming that the color composition of our results are more similar to Seurat's painting style than that of previous approaches. Additionally, we also conducted a user study participated by artists to qualitatively evaluate the synthesized images of the proposed method.     

Adding Depth to Cartoons Using Sparse Depth (In)equalities

This paper presents a novel interactive approach for adding depth information into hand‐drawn cartoon images and animations. In comparison to previous depth assignment techniques our solution requires minimal user effort and enables creation of consistent pop‐ups in a matter of seconds. Inspired by perceptual studies we formulate a custom tailored optimization framework that tries to mimic the way that a human reconstructs depth information from a single image. Its key advantage is that it completely avoids inputs requiring knowledge of absolute depth and instead uses a set of sparse depth (in)equalities that are much easier to specify. Since these constraints lead to a solution based on quadratic programming that is time consuming to evaluate we propose a simple approximative algorithm yielding similar results with much lower computational overhead. We demonstrate its usefulness in the context of a cartoon animation production pipeline including applications such as enhancement, registration, composition, 3D modelling and stereoscopic display.     

Modeling Complex Unfoliaged Trees from a Sparse Set of Images

We present a novel image‐based technique for modeling complex unfoliaged trees. Existing tree modeling tools either require capturing a large number of views for dense 3D reconstruction or rely on user inputs and botanic rules to synthesize natural‐looking tree geometry. In this paper, we focus on faithfully recovering real instead of realistically‐looking tree geometry from a sparse set of images. Our solution directly integrates 2D/3D tree topology as shape priors into the modeling process. For each input view, we first estimate a 2D skeleton graph from its matte image and then find a 2D skeleton tree from the graph by imposing tree topology. We develop a simple but effective technique for computing the optimal 3D skeleton tree most consistent with the 2D skeletons. For each edge in the 3D skeleton tree, we further apply volumetric reconstruction to recover its corresponding curved branch. Finally, we use piecewise cylinders to approximate each branch from the volumetric results. We demonstrate our framework on a variety of trees to illustrate the robustness and usefulness of our technique.     

Estimation and Modeling of Actual Numerical Errors in Volume Rendering

In this paper we study the comprehensive effects on volume rendered images due to numerical errors caused by the use of finite precision for data representation and processing. To estimate actual error behavior we conduct a thorough study using a volume renderer implemented with arbitrary floating‐point precision. Based on the experimental data we then model the impact of floating‐point pipeline precision, sampling frequency and fixed‐point input data quantization on the fidelity of rendered images. We introduce three models, an average model, which does not adapt to different data nor varying transfer functions, as well as two adaptive models that take the intricacies of a new data set and transfer function into account by adapting themselves given a few different images rendered. We also test and validate our models based on new data that was not used during our model building.     

An Example‐based Approach to Synthesize Artistic Strokes using Graphs

In this paper, we propose a technique to produce artistic strokes in a variety of drawing material based on example images. Our approach is to divide example strokes scanned from images into small pieces along their stroke directions and synthesize a novel stroke by rearranging them along a user specified curve. The visible quality of a synthesized stroke can be maintained by utilizing the connectivity information stored in a directed graph constructed in the preprocessing step. At run‐time, the graph is traversed to find a path best matching the user specification given as a curve and additional information. The results of our experiments shows that visually convincing strokes of various materials can be generated efficiently.     

Sketching MLS Image Deformations On the GPU

In this paper, we present an image editing tool that allows the user to deform images using a sketch‐based interface. The user simply sketches a set of source curves in the input image, and also some target curves that the source curves should be deformed to. Then the moving least squares (MLS) deformation technique [ [SMW06] ] is adapted to produce realistic deformations while satisfying the curves' positional constraints. We also propose a scheme to reduce image fold‐overs in MLS deformations. Our system has a very intuitive user interface, generates physically plausible deformations, and can be easily implemented on the GPU for real‐time performance.     

Interactive Image‐Guided Modeling of Extruded Shapes

A recent trend in interactive modeling of 3D shapes from a single image is designing minimal interfaces, and accompanying algorithms, for modeling a specific class of objects. Expanding upon the range of shapes that existing minimal interfaces can model, we present an interactive image‐guided tool for modeling shapes made up of extruded parts. An extruded part is represented by extruding a closed planar curve, called base, in the direction orthogonal to the base. To model each extruded part, the user only needs to sketch the projected base shape in the image. The main technical contribution is a novel optimization‐based approach for recovering the 3D normal of the base of an extruded object by exploring both geometric regularity of the sketched curve and image contents. We developed a convenient interface for modeling multi‐part shapes and a method for optimizing the relative placement of the parts. Our tool is validated using synthetic data and tested on real‐world images.     

Freehand HDR Imaging of Moving Scenes with Simultaneous Resolution Enhancement

Despite their high popularity, common high dynamic range (HDR) methods are still limited in their practical applicability: They assume that the input images are perfectly aligned, which is often violated in practise. Our paper does not only free the user from this unrealistic limitation, but even turns the missing alignment into an advantage: By exploiting the multiple exposures, we can create a super‐resolution image. The alignment step is performed by a modern energy‐based optic flow approach that takes into account the varying exposure conditions. Moreover, it produces dense displacement fields with subpixel precision. As a consequence, our approach can handle arbitrary complex motion patterns, caused by severe camera shake and moving objects. Additionally, it benefits from several advantages over existing strategies: (i) It is robust under outliers (noise, occlusions, saturation problems) and allows for sharp discontinuities in the displacement field. (ii) The alignment step neither requires camera calibration nor knowledge of the exposure times. (iii) It can be efficiently implemented on CPU and GPU architectures. After the alignment is performed, we use the obtained subpixel accurate displacement fields as input for an energy‐based, joint super‐resolution and HDR (SR‐HDR) approach. It introduces robust data terms and anisotropic smoothness terms in the SR‐HDR literature. Our experiments with challenging real world data demonstrate that these novelties are pivotal for the favourable performance of our approach.