Shrinkability Maps for Content‐Aware Video Resizing

A novel method is given for content‐aware video resizing, i.e. targeting video to a new resolution (which may involve aspect ratio change) from the original. We precompute a per‐pixel cumulative shrinkability map which takes into account both the importance of each pixel and the need for continuity in the resized result. (If both x and y resizing are required, two separate shrinkability maps are used, otherwise one suffices). A random walk model is used for efficient offline computation of the shrinkability maps. The latter are stored with the video to create a multi‐sized video, which permits arbitrary‐sized new versions of the video to be later very efficiently created in real‐time, e.g. by a video‐on‐demand server supplying video streams to multiple devices with different resolutions. These shrinkability maps are highly compressible, so the resulting multi‐sized videos are typically less than three times the size of the original compressed video. A scaling function operates on the multi‐sized video, to give the new pixel locations in the result, giving a high‐quality content‐aware resized video. Despite the great efficiency and low storage requirements for our method, we produce results of comparable quality to state‐of‐the‐art methods for content‐aware image and video resizing.     

Gradient‐Preserving Color Transfer

Color transfer is an image processing technique which can produce a new image combining one source image's contents with another image's color style. While being able to produce convincing results, however, Reinhard et al.'s pioneering work has two problems—mixing up of colors in different regions and the fidelity problem. Many local color transfer algorithms have been proposed to resolve the first problem, but the second problem was paid few attentions. In this paper, a novel color transfer algorithm is presented to resolve the fidelity problem of color transfer in terms of scene details and colors. It's well known that human visual system is more sensitive to local intensity differences than to intensity itself. We thus consider that preserving the color gradient is necessary for scene fidelity. We formulate the color transfer problem as an optimization problem and solve it in two steps—histogram matching and a gradient‐preserving optimization. Following the idea of the fidelity in terms of color and gradient, we also propose a metric for objectively evaluating the performance of example‐based color transfer algorithms. The experimental results show the validity and high fidelity of our algorithm and that it can be used to deal with local color transfer.     

Multiple Facial Image Editing Using Edge–Aware PDE Learning

This paper introduces a novel facial editing tool, called edge‐aware mask, to achieve multiple photo‐realistic rendering effects in a unified framework. The edge‐aware masks facilitate three basic operations for adaptive facial editing, including region selection, edit setting and region blending. Inspired by the state‐of‐the‐art edit propagation and partial differential equation (PDE) learning method, we propose an adaptive PDE model with facial priors for masks generation through edge‐aware diffusion. The edge‐aware masks can automatically fit the complex region boundary with great accuracy and produce smooth transition between different regions, which significantly improves the visual consistence of face editing and reduce the human intervention. Then, a unified and flexible facial editing framework is constructed, which consists of layer decomposition, edge‐aware masks generation, and layer/mask composition. The combinations of multiple facial layers and edge‐aware masks can achieve various facial effects simultaneously, including face enhancement, relighting, makeup and face blending etc. Qualitative and quantitative evaluations were performed using different datasets for different facial editing tasks. Experiments demonstrate the effectiveness and flexibility of our methods, and the comparisons with the previous methods indicate that improved results are obtained using the combination of multiple edge‐aware masks.     

Stroke‐guided Image Synthesis for Skeletal Structure Editing

Creating variations of an image object is an important task, which usually requires manipulating the skeletal structure of the object. However, most existing methods (such as image deformation) only allow for stretching the skeletal structure of an object: modifying skeletal topology remains a challenge. This paper presents a technique for synthesizing image objects with different skeletal structures while respecting to an input image object. To apply this technique, a user firstly annotates the skeletal structure of the input object by specifying a number of strokes in the input image, and draws corresponding strokes in an output domain to generate new skeletal structures. Then, a number of the example texture pieces are sampled along the strokes in the input image and pasted along the strokes in the output domain with their orientations. The result is obtained by optimizing the texture sampling and seam computation. The proposed method is successfully used to synthesize challenging skeletal structures, such as skeletal branches, and a wide range of image objects with various skeletal structures, to demonstrate its effectiveness.     

Semi‐automatic Vortex Flow Classification in 4D PC‐MRI Data of the Aorta

We present an Aortic Vortex Classification (AVOCLA) that allows to classify vortices in the human aorta semi‐automatically. Current medical studies assume a strong relation between cardiovascular diseases and blood flow patterns such as vortices. Such vortices are extracted and manually classified according to specific, unstandardized properties. We employ an agglomerative hierarchical clustering to group vortex‐representing path lines as basis for the subsequent classification. Classes are based on the vortex' size, orientation and shape, its temporal occurrence relative to the cardiac cycle as well as its spatial position relative to the vessel course. The classification results are presented by a 2D and 3D visualization technique. To confirm the usefulness of both approaches, we report on the results of a user study. Moreover, AVOCLA was applied to 15 datasets of healthy volunteers and patients with different cardiovascular diseases. The results of the semi‐automatic classification were qualitatively compared to a manually generated ground truth of two domain experts considering the vortex number and five specific properties.     

Glyph‐Based Comparative Stress Tensor Visualization in Cerebral Aneurysms

We present the first visualization tool that enables a comparative depiction of structural stress tensor data for vessel walls of cerebral aneurysms. Such aneurysms bear the risk of rupture, whereas their treatment also carries considerable risks for the patient. Medical researchers emphasize the importance of analyzing the interaction of morphological and hemodynamic information for the patient‐specific rupture risk evaluation and treatment analysis. Tensor data such as the stress inside the aneurysm walls characterizes the interplay between the morphology and blood flow and seems to be an important rupture‐prone criterion. We use different glyph‐based techniques to depict local stress tensors simultaneously and compare their applicability to cerebral aneurysms in a user study. We thus offer medical researchers an effective visual exploration tool to assess the aneurysm rupture risk. We developed a GPU‐based implementation of our techniques with a flexible interactive data exploration mechanism. Our depictions are designed in collaboration with domain experts, and we provide details about the evaluation.     

Optimizing Structure Preserving Embedded Deformation for Resizing Images and Vector Art

Smart deformation and warping tools play an important part in modern day geometric modeling systems. They allow existing content to be stretched or scaled while preserving visually salient information. To date, these techniques have primarily focused on preserving local shape details, not taking into account important global structures such as symmetry and line features. In this work we present a novel framework that can be used to preserve the global structure in images and vector art. Such structures include symmetries and the spatial relations in shapes and line features in an image. Central to our method is a new formulation of preserving structure as an optimization problem. We use novel optimization strategies to achieve the interactive performance required by modern day modeling applications. We demonstrate the effectiveness of our framework by performing structure preservation deformation of images and complex vector art at interactive rates.     

A Graph‐based Approach to Continuous Line Illustrations with Variable Levels of Detail

This paper introduces a method for automatically generating continuous line illustrations, drawings consisting of a single line, from a given input image. Our approach begins by inferring a graph from a set of edges extracted from the image in question and obtaining a path that traverses through all edges of the said graph. The resulting path is then subjected to a series of post‐processing operations to transform it into a continuous line drawing. Moreover, our approach allows us to manipulate the amount of detail portrayed in our line illustrations, which is particularly useful for simplifying the overall illustration while still retaining its most significant features. We also present several experimental results to demonstrate that our approach can automatically synthesize continuous line illustrations comparable to those of some contemporary artists.     

Digital Camouflage Images Using Two‐scale Decomposition

We present an alternative approach to create digital camouflage images which follows human's perception intuition and complies with the physical creation procedure of artists. Our method is based on a two‐scale decomposition scheme of the input images. We modify the large‐scale layer of the background image by considering structural importance based on energy optimization and the detail layer by controlling its spatial variation. A gradient correction is presented to prevent halo artifacts. Users can control the difficulty level of perceiving the camouflage effect through a few parameters. Our camouflage images are natural and have less long coherent edges in the hidden region. Experimental results show that our algorithm yields visually pleasing camouflage images.     

Scale‐aware Structure‐Preserving Texture Filtering

This paper presents a novel method to enhance the performance of structure‐preserving image and texture filtering. With conventional edge‐aware filters, it is often challenging to handle images of high complexity where features of multiple scales coexist. In particular, it is not always easy to find the right balance between removing unimportant details and protecting important features when they come in multiple sizes, shapes, and contrasts. Unlike previous approaches, we address this issue from the perspective of adaptive kernel scales. Relying on patch‐based statistics, our method identifies texture from structure and also finds an optimal per‐pixel smoothing scale. We show that the proposed mechanism helps achieve enhanced image/texture filtering performance in terms of protecting the prominent geometric structures in the image, such as edges and corners, and keeping them sharp even after significant smoothing of the original signal.     

ScribbleBoost: Adding Classification to Edge‐Aware Interpolation of Local Image and Video Adjustments

One of the most common tasks in image and video editing is the local adjustment of various properties (e.g., saturation or brightness) of regions within an image or video. Edge‐aware interpolation of user‐drawn scribbles offers a less effort‐intensive approach to this problem than traditional region selection and matting. However, the technique suffers a number of limitations, such as reduced performance in the presence of texture contrast, and the inability to handle fragmented appearances. We significantly improve the performance of edge‐aware interpolation for this problem by adding a boosting‐based classification step that learns to discriminate between the appearance of scribbled pixels. We show that this novel data term in combination with an existing edge‐aware optimization technique achieves substantially better results for the local image and video adjustment problem than edge‐aware interpolation techniques without classification, or related methods such as matting techniques or graph cut segmentation.     

High‐Order Recursive Filtering of Non‐Uniformly Sampled Signals for Image and Video Processing

We present a discrete‐time mathematical formulation for applying recursive digital filters to non‐uniformly sampled signals. Our solution presents several desirable features: it preserves the stability of the original filters; is well‐conditioned for low‐pass, high‐pass, and band‐pass filters alike; its cost is linear in the number of samples and is not affected by the size of the filter support. Our method is general and works with any non‐uniformly sampled signal and any recursive digital filter defined by a difference equation. Since our formulation directly uses the filter coefficients, it works out‐of‐the‐box with existing methodologies for digital filter design. We demonstrate the effectiveness of our approach by filtering non‐uniformly sampled signals in various image and video processing tasks including edge‐preserving color filtering, noise reduction, stylization, and detail enhancement. Our formulation enables, for the first time, edge‐aware evaluation of any recursive infinite impulse response digital filter (not only low‐pass), producing high‐quality filtering results in real time.     

A Hidden‐picture Puzzles Generator

A hidden‐picture puzzle contains objects hidden in a background image, in such a way that each object fits closely into a local region of the background. Our system converts image of the background and objects into line drawing, and then finds places in which to hide transformed versions of the objects using rotation‐invariant shape context matching. During the hiding process, each object is subjected to a slight deformation to enhance its similarity to the background. The results were assessed by a panel of puzzle‐solvers.     

Feature‐Preserving Reconstruction of Singular Surfaces

Reconstructing a surface mesh from a set of discrete point samples is a fundamental problem in geometric modeling. It becomes challenging in presence of ‘singularities’ such as boundaries, sharp features, and non‐manifolds. A few of the current research in reconstruction have addressed handling some of these singularities, but a unified approach to handle them all is missing. In this paper we allow the presence of various singularities by requiring that the sampled object is a collection of smooth surface patches with boundaries that can meet or intersect. Our algorithm first identifies and reconstructs the features where singularities occur. Next, it reconstructs the surface patches containing these feature curves. The identification and reconstruction of feature curves are achieved by a novel combination of the Gaussian weighted graph Laplacian and the Reeb graphs. The global reconstruction is achieved by a method akin to the well known Cocone reconstruction, but with weighted Delaunay triangulation that allows protecting the feature samples with balls. We provide various experimental results to demonstrate the effectiveness of our feature‐preserving singular surface reconstruction algorithm.     

Visual and quantitative analysis of great arteries’ blood flow jets in cardiac 4D PC‐MRI data

Flow in the great arteries (aorta, pulmonary artery) is normally laminar with a parabolic velocity profile. Eccentric flow jets are linked to various diseases like aneurysms. Cardiac 4D PC‐MRI data provide spatio‐temporally resolved blood flow information for the whole cardiac cycle. In this work, we establish a time‐dependent visualization and quantification of flow jets. For this purpose, equidistant measuring planes are automatically placed along the vessel's centerline. The flow jet position and region with highest velocities are extracted for every plane in each time step. This is done during pre‐processing and without user‐defined parameters. We visualize the main flow jet as geometric tube. High‐velocity areas are depicted as a net around this tube. Both geometries are time‐dependent and can be animated. Quantitative values are provided during cross‐sectional measuring plane‐based evaluation. Moreover, we offer a plot visualization as summary of flow jet characteristics for the selected plane. Our physiologically plausible results are in accordance with medical findings. Our clinical collaborators appreciate the possibility to view the flow jet in the whole vessel at once, which normally requires repeated pathline filtering due to varying velocities along the vessel course. The overview plots are considered as valuable for documentation purposes.     

Capturing Intention‐based Full‐Frame Video Stabilization

Annoying shaky motion is one of the significant problems in home videos, since hand shake is an unavoidable effect when capturing by using a hand‐held camcorder. Video stabilization is an important technique to solve this problem, but the stabilized videos resulting from some current methods usually have decreased resolution and are still not so stable. In this paper, we propose a robust and practical method of full‐frame video stabilization while considering user's capturing intention to remove not only the high frequency shaky motions but also the low frequency unexpected movements. To guess the user's capturing intention, we first consider the regions of interest in the video to estimate which regions or objects the user wants to capture, and then use a polyline to estimate a new stable camcorder motion path while avoiding the user's interested regions or objects being cut out. Then, we fill the dynamic and static missing areas caused by frame alignment from other frames to keep the same resolution and quality as the original video. Furthermore, we smooth the discontinuous regions by using a three‐dimensional Poisson‐based method. After the above automatic operations, a full‐frame stabilized video can be achieved and the important regions and objects can also be preserved.     

Shape‐simplifying Image Abstraction

This paper presents a simple algorithm for producing stylistic abstraction of a photograph. Based on mean curvature flow in conjunction with shock filter, our method simplifies both shapes and colors simultaneously while preserving important features. In particular, we develop a constrained mean curvature flow, which outperforms the original mean curvature flow in conveying the directionality of features and shape boundaries. The proposed algorithm is iterative and incremental, and therefore the level of abstraction is intuitively controlled. Optionally, simple user masking can be incorporated into the algorithm to selectively control the abstraction speed and to protect particular regions. Experimental results show that our method effectively produces highly abstract yet feature‐preserving illustrations from photographs.     

Jointly Optimized Regressors for Image Super‐resolution

Learning regressors from low‐resolution patches to high‐resolution patches has shown promising results for image super‐resolution. We observe that some regressors are better at dealing with certain cases, and others with different cases. In this paper, we jointly learn a collection of regressors, which collectively yield the smallest super‐resolving error for all training data. After training, each training sample is associated with a label to indicate its ‘best’ regressor, the one yielding the smallest error. During testing, our method bases on the concept of ‘adaptive selection’ to select the most appropriate regressor for each input patch. We assume that similar patches can be super‐resolved by the same regressor and use a fast, approximate kNN approach to transfer the labels of training patches to test patches. The method is conceptually simple and computationally efficient, yet very effective. Experiments on four datasets show that our method outperforms competing methods.     

Parallel Blue‐noise Sampling by Constrained Farthest Point Optimization

We describe a fast sampling algorithm for generating uniformly‐distributed point patterns with good blue noise characteristics. The method, based on constrained farthest point optimization, is provably optimal and may be easily parallelized, resulting in an algorithm whose performance/quality tradeoff is superior to other state‐of‐the‐art approaches.     

Image Dequantization: Restoration of Quantized Colors

Color quantization replaces the color of each pixel with the closest representative color, and thus it makes the resulting image partitioned into uniformly-colored regions. As a consequence, continuous, detailed variations of color over the corresponding regions in the original image are lost through color quantization. In this paper, we present a novel blind scheme for restoring such variations from a color-quantized input image without a priori knowledge of the quantization method. Our scheme identifies which pairs of uniformly-colored regions in the input image should have continuous variations of color in the resulting image. Then, such regions are seamlessly stitched through optimization while preserving the closest representative colors. The user can optionally indicate which regions should be separated or stitched by scribbling constraint brushes across the regions. We demonstrate the effectiveness of our approach through diverse examples, such as photographs, cartoons, and artistic illustrations.