Hardware‐Based Non‐Photorealistic Rendering Using a Painting Robot

We describe a painting machine and associated algorithms. Our modified industrial robot works with visual feedback and applies acrylic paint from a repository to a canvas until the created painting resembles a given input image or scene. The color differences between canvas and input are used to direct the application of new strokes. We present two optimization‐based algorithms that place such strokes in relation to already existing ones. Using these methods we are able to create different painting styles, one that tries to match the input colors with almost transparent strokes and another one that creates dithering patterns of opaque strokes that approximate the input color. The machine produces paintings that mimic those created by human painters and allows us to study the painting process as well as the creation of artworks.     

Perception‐driven Accelerated Rendering

Advances in computer graphics enable us to create digital images of astonishing complexity and realism. However, processing resources are still a limiting factor. Hence, many costly but desirable aspects of realism are often not accounted for, including global illumination, accurate depth of field and motion blur, spectral effects, etc. especially in real‐time rendering. At the same time, there is a strong trend towards more pixels per display due to larger displays, higher pixel densities or larger fields of view. Further observable trends in current display technology include more bits per pixel (high dynamic range, wider color gamut/fidelity), increasing refresh rates (better motion depiction), and an increasing number of displayed views per pixel (stereo, multi‐view, all the way to holographic or lightfield displays). These developments cause significant unsolved technical challenges due to aspects such as limited compute power and bandwidth. Fortunately, the human visual system has certain limitations, which mean that providing the highest possible visual quality is not always necessary. In this report, we present the key research and models that exploit the limitations of perception to tackle visual quality and workload alike. Moreover, we present the open problems and promising future research targeting the question of how we can minimize the effort to compute and display only the necessary pixels while still offering a user full visual experience.     

Minimum‐Displacement Overlap Removal for Geo‐referenced Data Visualization

Given a set of rectangles embedded in the plane, we consider the problem of adjusting the layout to remove all overlap while preserving the orthogonal order of the rectangles. The objective is to minimize the displacement of the rectangles. We call this problem Minimum -Displacement Overlap Removal (mdor ). Our interest in this problem is motivated by the application of displaying metadata of archaeological sites. Because most existing overlap removal algorithms are not designed to minimize displacement while preserving orthogonal order, we present and compare several approaches which are tailored to our particular usecase. We introduce a new overlap removal heuristic which we call re Arrange . Although conceptually simple, it is very effective in removing the overlap while keeping the displacement small. Furthermore, we propose an additional procedure to repair the orthogonal order after every iteration, with which we extend both our new heuristic and PRISM, a widely used overlap removal algorithm. We compare the performance of both approaches with and without this order repair method. The experimental results indicate that re Arrange is very effective for heterogeneous input data where the overlap is concentrated in few dense regions.     

Rule‐Enhanced Transfer Function Generation for Medical Volume Visualization

In volume visualization, transfer functions are used to classify the volumetric data and assign optical properties to the voxels. In general, transfer functions are generated in a transfer function space, which is the feature space constructed by data values and properties derived from the data. If volumetric objects have the same or overlapping data values, it would be difficult to separate them in the transfer function space. In this paper, we present a rule‐enhanced transfer function design method that allows important structures of the volume to be more effectively separated and highlighted. We define a set of rules based on the local frequency distribution of volume attributes. A rule‐selection method based on a genetic algorithm is proposed to learn the set of rules that can distinguish the user‐specified target tissue from other tissues. In the rendering stage, voxels satisfying these rules are rendered with higher opacities in order to highlight the target tissue. The proposed method was tested on various volumetric datasets to enhance the visualization of important structures that are difficult to be visualized by traditional transfer function design methods. The results demonstrate the effectiveness of the proposed method.     

A Survey of Perceptually Motivated 3D Visualization of Medical Image Data

This survey provides an overview of perceptually motivated techniques for the visualization of medical image data, including physics‐based lighting techniques as well as illustrative rendering that incorporate spatial depth and shape cues. Additionally, we discuss evaluations that were conducted in order to study the perceptual effects of these visualization techniques as compared to conventional techniques. These evaluations assessed depth and shape perception with depth judgment, orientation matching, and related tasks. This overview of existing techniques and their evaluation serves as a basis for defining the evaluation process of medical visualizations and to discuss a research agenda.     

Real‐time Texture Synthesis and Concurrent Random‐access Rendering for Low‐cost GPU Chip Design

Numerous algorithms have been researched in the area of texture synthesis. However, it remains difficult to design a low‐cost synthesis scheme capable of generating high quality results while simultaneously achieving real‐time performance. Additional challenges include making a scheme parallel and being able to partially render/synthesize high‐resolution textures. Furthermore, it would be beneficial for a synthesis scheme to be able to incorporate Texture Compression and minimize the bandwidth usage, especially on mobile devices. In this paper, we propose a practical method which has low computational complexity and produces textures with small storage requirements. Through use of an index table, random access of the texture is another essential advantage, with which parallel rendering becomes feasible including generation of mip‐map sequences. Integrating the index table with existing compression algorithms, for example ETC or PVRTC, the bandwidth is further reduced and avoids the need for a separate, computationally expensive pass to compress the synthesized output. It should be noted that our texture synthesis achieves real‐time performance and low power consumption even on mobile devices, for which texture synthesis has been traditionally considered too expensive.     

Field‐aligned Quadrangulation for Image Vectorization

Image vectorization is an important yet challenging problem, especially when the input image has rich content. In this paper, we develop a novel method for automatically vectorizing natural images with feature‐aligned quad‐dominant meshes. Inspired by the quadrangulation methods in 3D geometry processing, we propose a new directional field optimization technique by encoding the color gradients, sidestepping the explicit computing of salient image features. We further compute the anisotropic scales of the directional field by accommodating the distance among image features. Our method is fully automatic and efficient, which takes only a few seconds for a 400×400 image on a normal laptop. We demonstrate the effectiveness of the proposed method on various image editing applications.     

Compressive Volume Rendering

Compressive rendering refers to the process of reconstructing a full image from a small subset of the rendered pixels, thereby expediting the rendering task. In this paper, we empirically investigate three image order techniques for compressive rendering that are suitable for direct volume rendering. The first technique is based on the theory of compressed sensing and leverages the sparsity of the image gradient in the Fourier domain. The latter techniques exploit smoothness properties of the rendered image; the second technique recovers the missing pixels via a total variation minimization procedure while the third technique incorporates a smoothness prior in a variational reconstruction framework employing interpolating cubic B‐splines. We compare and contrast the three techniques in terms of quality, efficiency and sensitivity to the distribution of pixels. Our results show that smoothness‐based techniques significantly outperform techniques that are based on compressed sensing and are also robust in the presence of highly incomplete information. We achieve high quality recovery with as little as 20% of the pixels distributed uniformly in screen space.     

Regularizing Image Reconstruction for Gradient‐Domain Rendering with Feature Patches

We present a novel algorithm to reconstruct high‐quality images from sampled pixels and gradients in gradient‐domain Rendering. Our approach extends screened Poisson reconstruction by adding additional regularization constraints. Our key idea is to exploit local patches in feature images, which contain per‐pixels normals, textures, position, etc., to formulate these constraints. We describe a GPU implementation of our approach that runs on the order of seconds on megapixel images. We demonstrate a significant improvement in image quality over screened Poisson reconstruction under the L₁ norm. Because we adapt the regularization constraints to the noise level in the input, our algorithm is consistent and converges to the ground truth.     

Adaptive multi‐scale analysis for point‐based surface editing

This paper presents a tool that enables the direct editing of surface features in large point‐clouds or meshes. This is made possible by a novel multi‐scale analysis of unstructured point‐clouds that automatically extracts the number of relevant features together with their respective scale all over the surface. Then, combining this ingredient with an adequate multi‐scale decomposition allows us to directly enhance or reduce each feature in an independent manner. Our feature extraction is based on the analysis of the scale‐variations of locally fitted surface primitives combined with unsupervised learning techniques. Our tool may be applied either globally or locally, and millions of points are handled in real‐time. The resulting system enables users to accurately edit complex geometries with minimal interaction.     

Sub‐Pixel Anti‐Aliasing Via Triangle‐Based Geometry Reconstruction

Anti‐aliasing has recently been employed as a post‐processing step to adapt to the deferred shading technique in real‐time applications. Some of these existing algorithms store supersampling geometric information as geometric buffer (G‐buffer) to detect and alleviate sub‐pixel‐level aliasing artifacts. However, the anti‐aliasing filter based on sampled sub‐pixel geometries only may introduce unfaithful shading information to the sub‐pixel color in uniform‐geometry regions, and large G‐buffer will increase memory storage and fetch overheads. In this paper, we present a new Triangle‐based Geometry Anti‐Aliasing (TGAA) algorithm, to address these problems. The coverage triangle of each screen pixel is accessed, and then, the coverage information between the triangle and neighboring sub‐pixels is stored in a screen‐resolution bitmask, which allows the geometric information to be stored and accessed in an inexpensive manner. Using triangle‐based geometry, TGAA can exclude irrelevant neighboring shading samples and achieve faithful anti‐aliasing filtering. In addition, a morphological method of estimating the geometric edges in high‐frequency geometry is incorporated into the TGAA's anti‐aliasing filter to complement the algorithm. The implementation results demonstrate that the algorithm is efficient and scalable for generating high‐quality anti‐aliased images.     

Selective Degree Elevation for Multi‐Sided Bézier Patches

This paper presents a method to selectively elevate the degree of an S‐Patch of arbitrary dimension. We consider not only S‐Patches with 2D domains but 3D and higher‐dimensional domains as well, of which volumetric cage deformations are a subset. We show how to selectively insert control points of a higher degree patch into a lower degree patch while maintaining the polynomial reproduction order of the original patch. This process allows the user to elevate the degree of only one portion of the patch to add new degrees of freedom or maintain continuity with adjacent patches without elevating the degree of the entire patch, which could create far more degrees of freedom than necessary. Finally we show an application to cage‐based deformations where we increase the number of control points by elevating the degree of a subset of cage faces. The result is a cage deformation with higher degree triangular Bézier functions on a subset of cage faces but no interior control points.     

Multiscale Visualization and Exploration of Large Bipartite Graphs

A bipartite graph is a powerful abstraction for modeling relationships between two collections. Visualizations of bipartite graphs allow users to understand the mutual relationships between the elements in the two collections, e.g., by identifying clusters of similarly connected elements. However, commonly‐used visual representations do not scale for the analysis of large bipartite graphs containing tens of millions of vertices, often resorting to an a‐priori clustering of the sets. To address this issue, we present the Who's‐Active‐On‐What‐Visualization (WAOW‐Vis) that allows for multiscale exploration of a bipartite social‐network without imposing an a‐priori clustering. To this end, we propose to treat a bipartite graph as a high‐dimensional space and we create the WAOW‐Vis adapting the multiscale dimensionality‐reduction technique HSNE. The application of HSNE for bipartite graph requires several modifications that form the contributions of this work. Given the nature of the problem, a set‐based similarity is proposed. For efficient and scalable computations, we use compressed bitmaps to represent sets and we present a novel space partitioning tree to efficiently compute similarities; the Sets Intersection Tree. Finally, we validate WAOW‐Vis on several datasets connecting Twitter‐users and ‐streams in different domains: news, computer science and politics. We show how WAOW‐Vis is particularly effective in identifying hierarchies of communities among social‐media users.     

High‐resolution 360 Video Foveated Stitching for Real‐time VR

In virtual reality (VR) applications, the contents are usually generated by creating a 360° Video panorama of a real‐world scene. Although many capture devices are being released, getting high‐resolution panoramas and displaying a virtual world in real‐time remains challenging due to its computationally demanding nature. In this paper, we propose a real‐time 360° Video foveated stitching framework, that renders the entire scene in different level of detail, aiming to create a high‐resolution panoramic Video in real‐time that can be streamed directly to the client. Our foveated stitching algorithm takes Videos from multiple cameras as input, combined with measurements of human visual attention (i.e. the acuity map and the saliency map), can greatly reduce the number of pixels to be processed. We further parallelize the algorithm using GPU to achieve a responsive interface and validate our results via a user study. Our system accelerates graphics computation by a factor of 6 on a Google Cardboard display.     

Time‐Series Plots Integrated in Parallel‐Coordinates Displays

We present a natural extension of two‐dimensional parallel‐coordinates plots for revealing relationships in time‐dependent multi‐attribute data by building on the idea that time can be considered as the third dimension. A time slice through the visualization represents a certain point in time and can be viewed as a regular parallel‐coordinates display. A vertical slice through one of the axes of the parallel‐coordinates display would show a time‐series plot. For a focus‐and‐context Integration of both views, we embed time‐series plots between two adjacent axes of the parallel‐coordinates plot. Both time‐series plots are drawn using a pseudo three‐dimensional perspective with a single vanishing point. An independent parallel‐coordinates panel that connects the two perspectively displayed time‐series plots can move forward and backward in time to reveal changes in the relationship between the time‐dependent attributes. The visualization of time‐series plots in the context of the parallel‐coordinates plot facilitates the exploration of time‐related aspects of the data without the need to switch to a separate display. We provide a consistent set of tools for selecting and contrasting subsets of the data, which are important for various application domains.     

Sliceplorer: 1D slices for multi‐dimensional continuous functions

Multi‐dimensional continuous functions are commonly visualized with 2D slices or topological views. Here, we explore 1D slices as an alternative approach to show such functions. Our goal with 1D slices is to combine the benefits of topological views, that is, screen space efficiency, with those of slices, that is a close resemblance of the underlying function. We compare 1D slices to 2D slices and topological views, first, by looking at their performance with respect to common function analysis tasks. We also demonstrate 3 usage scenarios: the 2D sinc function, neural network regression, and optimization traces. Based on this evaluation, we characterize the advantages and drawbacks of each of these approaches, and show how interaction can be used to overcome some of the shortcomings.     

The Grassmannian Atlas: A General Framework for Exploring Linear Projections of High‐Dimensional Data

Linear projections are one of the most common approaches to visualize high‐dimensional data. Since the space of possible projections is large, existing systems usually select a small set of interesting projections by ranking a large set of candidate projections based on a chosen quality measure. However, while highly ranked projections can be informative, some lower ranked ones could offer important complementary information. Therefore, selection based on ranking may miss projections that are important to provide a global picture of the data. The proposed work fills this gap by presenting the Grassmannian Atlas, a framework that captures the global structures of quality measures in the space of all projections, which enables a systematic exploration of many complementary projections and provides new insights into the properties of existing quality measures.     

Flow‐Induced Inertial Steady Vector Field Topology

Traditionally, vector field visualization is concerned with 2D and 3D flows. Yet, many concepts can be extended to general dynamical systems, including the higher‐dimensional problem of modeling the motion of finite‐sized objects in fluids. In the steady case, the trajectories of these so‐called inertial particles appear as tangent curves of a 4D or 6D vector field. These higher‐dimensional flows are difficult to map to lower‐dimensional spaces, which makes their visualization a challenging problem. We focus on vector field topology, which allows scientists to study asymptotic particle behavior. As recent work on the 2D case has shown, both extraction and classification of isolated critical points depend on the underlying particle model. In this paper, we aim for a model‐independent classification technique, which we apply to two different particle models in not only 2D, but also 3D cases. We show that the classification can be done by performing an eigenanalysis of the spatial derivatives' velocity subspace of the higher‐dimensional 4D or 6D flow. We construct glyphs that depict not only the types of critical points, but also encode the directional information given by the eigenvectors. We show that the eigenvalues and eigenvectors of the inertial phase space have sufficient symmetries and structure so that they can be depicted in 2D or 3D, instead of 4D or 6D.     

Information Visualization Evaluation Using Crowdsourcing

Visualization researchers have been increasingly leveraging crowdsourcing approaches to overcome a number of limitations of controlled laboratory experiments, including small participant sample sizes and narrow demographic backgrounds of study participants. However, as a community, we have little understanding on when, where, and how researchers use crowdsourcing approaches for visualization research. In this paper, we review the use of crowdsourcing for evaluation in visualization research. We analyzed 190 crowdsourcing experiments, reported in 82 papers that were published in major visualization conferences and journals between 2006 and 2017. We tagged each experiment along 36 dimensions that we identified for crowdsourcing experiments. We grouped our dimensions into six important aspects: study design & procedure, task type, participants, measures & metrics, quality assurance, and reproducibility. We report on the main findings of our review and discuss challenges and opportunities for improvements in conducting crowdsourcing studies for visualization research.