Virtual Viewpoint Replay for a Soccer Match by View Interpolation From Multiple Cameras

This paper presents a novel method for virtual view synthesis that allows viewers to virtually fly through real soccer scenes, which are captured by multiple cameras in a stadium. The proposed method generates images of arbitrary viewpoints by view interpolation of real camera images near the chosen viewpoints. In this method, cameras do not need to be strongly calibrated since projective geometry between cameras is employed for the interpolation. For avoiding the complex and unreliable process of 3-D recovery, object scenes are segmented into several regions according to the geometric property of the scene. Dense correspondence between real views, which is necessary for intermediate view generation, is automatically obtained by applying projective geometry to each region. By superimposing intermediate images for all regions, virtual views for the entire soccer scene are generated. The efforts for camera calibration are reduced and correspondence matching requires no manual operation; hence, the proposed method can be easily applied to dynamic events in a large space. An application for fly-through observations of soccer match replays is introduced along with the algorithm of view synthesis and experimental results. This is a new approach for providing arbitrary views of an entire dynamic event.     

Methods for Volumetric Reconstruction of Visual Scenes

In this paper, we present methods for 3D volumetric reconstruction of visual scenes photographed by multiple calibrated cameras placed at arbitrary viewpoints. Our goal is to generate a 3D model that can be rendered to synthesize new photo-realistic views of the scene. We improve upon existing voxel coloring/space carving approaches by introducing new ways to compute visibility and photo-consistency, as well as model infinitely large scenes. In particular, we describe a visibility approach that uses all possible color information from the photographs during reconstruction, photo-consistency measures that are more robust and/or require less manual intervention, and a volumetric warping method for application of these reconstruction methods to large-scale scenes.     

Efficient Algorithms for Computing Conservative Portal Visibility Information

The number of polygons in realistic architectural models is many more than can be rendered at interactive frame rates. Typically, however, due to occlusion by opaque surfaces (e.g., walls), only small fractions of such models are visible from most viewpoints. This fact is used in many popular methods for preprocessing visibility information which assume a scene model subdivided into convex cells connected through convex portals. These methods need to establish which cells or parts thereof are visible to a generalized observer located within each cell. The geometry of this information is termed a 'visibility volume' and its computation is usually quite complex. Conservative approximations of viewing volumes, however, are simpler and less expensive to compute. In this paper we present techniques and algorithms which permit the computation of conservative viewing volumes incrementally. In particular, we describe an algorithm for computing the viewing volumes for a given cell through a sequence of ' m ' portals containing a total of ' n ' edges in O m n time.     

Probabilistic Visibility Evaluation for Direct Illumination

The efficient evaluation of visibility in a three‐dimensional scene is a longstanding problem in computer graphics. Visibility evaluations come in many different forms: figuring out what object is visible in a pixel; determining whether a point is visible to a light source; or evaluating the mutual visibility between 2 surface points. This paper provides a new, experimental view on visibility, based on a probabilistic evaluation of the visibility function. Instead of checking the visibility against all possible intervening geometry the visibility between 2 points is now evaluated by testing only a random subset of objects. The result is not a Boolean value that is either 0 or 1, but a numerical value that can even be negative. Because we use the visibility evaluation as part of the integrand in illumination computations, the probabilistic evaluation of visibility becomes part of the Monte Carlo procedure of estimating the illumination integral, and results in an unbiased computation of illumination values in the scene. Moreover, the number of intersections tests for any given ray is decreased, since only a random selection of geometric primitives is tested. Although probabilistic visibility is an experimental and new idea, we present a practical algorithm for direct illumination that uses the probabilistic nature of visibility evaluations.     

Appearance-Cloning: Photo-Consistent Scene Recovery from Multi-View Images

This paper introduces the novel volumetric methodology “appearance-cloning” as a viable solution for achieving a more improved photo-consistent scene recovery, including a greatly enhanced geometric recovery performance, from a set of photographs taken at arbitrarily distributed multiple camera viewpoints. We do so while solving many of the problems associated with previous stereo-based and volumetric methodologies. We redesign the photo-consistency decision problem of individual voxel in volumetric space as the photo-consistent shape search problem in image space, by generalizing the concept of the point correspondence search between two images in stereo-based approach, within a volumetric framework. In detail, we introduce a self-constrained greedy-style optimization methodology, which iteratively searches a more photo-consistent shape based on the probabilistic shape photo-consistency measure, by using the probabilistic competition between candidate shapes. Our new measure is designed to bring back the probabilistic photo-consistency of a shape by comparing the appearances captured from multiple cameras with those rendered from that shape using the per-pixel Maxwell model in image space. Through various scene recoveries experiments including specular and dynamic scenes, we demonstrate that if sufficient appearances are given enough to reflect scene characteristics, our appearance-cloning approach can successfully recover both the geometry and photometry information of a scene without any kind of scene-dependent algorithm tuning.     

New Visual Invariants for Terrain Navigation Without 3D Reconstruction

For autonomous vehicles to achieve terrain navigation, obstacles must be discriminated from terrain before any path planning and obstacle avoidance activity is undertaken. In this paper, a novel approach to obstacle detection has been developed. The method finds obstacles in the 2D image space, as opposed to 3D reconstructed space, using optical flow. Our method assumes that both nonobstacle terrain regions, as well as regions with obstacles, will be visible in the imagery. Therefore, our goal is to discriminate between terrain regions with obstacles and terrain regions without obstacles. Our method uses new visual linear invariants based on optical flow. Employing the linear invariance property, obstacles can be directly detected by using reference flow lines obtained from measured optical flow. The main features of this approach are: (1) 2D visual information (i.e., optical flow) is directly used to detect obstacles; no range, 3D motion, or 3D scene geometry is recovered; (2) knowledge about the camera-to-ground coordinate transformation is not required; (3) knowledge about vehicle (or camera) motion is not required; (4) the method is valid for the vehicle (or camera) undergoing general six-degree-of-freedom motion; (5) the error sources involved are reduced to a minimum, because the only information required is one component of optical flow. Numerous experiments using both synthetic and real image data are presented. Our methods are demonstrated in both ground and air vehicle scenarios.     

Depth image‐based rendering for multiview generation

Abstract— Techniques for 3‐D display have evolved from stereoscopic 3‐D systems to multiview 3‐D systems, which provide images corresponding to different viewpoints. Currently, new technology is required for application in multiview display systems that use input‐source formats such as 2‐D images to generate virtual‐view images of multiple viewpoints. Due to the changes in viewpoints, occlusion regions of the original image become disoccluded, resulting in problems related to the restoration of output image information that is not contained in the input image. In this paper, a method for generating multiview images through a two‐step process is proposed: (1) depth‐map refinement and (2) disoccluded‐area estimation and restoration. The first step, depth‐map processing, removes depth‐map noise, compensates for mismatches between RGB and depth, and preserves the boundaries and object shapes. The second step, disoccluded‐area estimation and restoration, predicts the disoccluded area by using disparity and restores information about the area by using information about neighboring frames that are most similar to the occlusion area. Finally, multiview rendering generates virtual‐view images by using a directional rendering algorithm with boundary blending.     

3-D Reconstruction of Shaded Objects from Multiple Images Under Unknown Illumination

We propose a variational algorithm to jointly estimate the shape, albedo, and light configuration of a Lambertian scene from a collection of images taken from different vantage points. Our work can be thought of as extending classical multi-view stereo to cases where point correspondence cannot be established, or extending classical shape from shading to the case of multiple views with unknown light sources. We show that a first naive formalization of this problem yields algorithms that are numerically unstable, no matter how close the initialization is to the true geometry. We then propose a computational scheme to overcome this problem, resulting in provably stable algorithms that converge to (local) minima of the cost functional. We develop a new model that explicitly enforces positivity in the light sources with the assumption that the object is Lambertian and its albedo is piecewise constant and show that the new model significantly improves the accuracy and robustness relative to existing approaches.     

Nonredundancy regularization based nonnegative matrix factorization with manifold learning for multiview data representation

In the real world, one object is usually described via multiple views or modalities. Many existing multiview clustering methods fuse the information of multiple views by learning a consensus representation. However, the feature learned in this manner is usually redundant and has neglected the distinctions among the different views. Addressing this issue, a method named nonredundancy regularization based nonnegative matrix factorization with manifold learning (NRRNMF-ML) is proposed in the paper. A novel nonredundancy regularizer defined with the Hilbert–Schmidt Independence Criterion (HSIC) is incorporated in the objective function of the proposed method. By minimizing this term, the redundant information among the multiple views can be effectively reduced and the distinct contributions of the different views can be encouraged. To further utilizing manifold structure information of the data, a manifold regularizer is also constructed and included in the objective function of the proposed method. For the proposed method, an iterative optimization strategy was designed to solve the problem; the corresponding proof is presented both theoretically and experimentally in this paper. Experimental results on five multiview data sets compared with several representative multiview clustering methods revealed the effectiveness of the proposed method.     

Tracking Multiple Occluding People by Localizing on Multiple Scene Planes

Occlusion and lack of visibility in crowded and cluttered scenes make it difficult to track individual people correctly and consistently, particularly in a single view. We present a multi-view approach to solving this problem. In our approach we neither detect nor track objects from any single camera or camera pair; rather evidence is gathered from all the cameras into a synergistic framework and detection and tracking results are propagated back to each view. Unlike other multi-view approaches that require fully calibrated views our approach is purely image-based and uses only 2D constructs. To this end we develop a planar homographic occupancy constraint that fuses foreground likelihood information from multiple views, to resolve occlusions and localize people on a reference scene plane. For greater robustness this process is extended to multiple planes parallel to the reference plane in the framework of plane to plane homologies. Our fusion methodology also models scene clutter using the Schmieder and Weathersby clutter measure, which acts as a confidence prior, to assign higher fusion weight to views with lesser clutter. Detection and tracking are performed simultaneously by graph cuts segmentation of tracks in the space-time occupancy likelihood data. Experimental results with detailed qualitative and quantitative analysis, are demonstrated in challenging multi-view, crowded scenes.     

Partitioning Trimmed Spline Surfaces into NonSelf-Occluding Regions for Visibility Computation

Computing the visible portions of curved surfaces from a given viewpoint is of great interest in many applications. It is closely related to the hidden surface removal problem in computer graphics, and machining applications in manufacturing. Most of the early work has focused on discrete methods based on polygonization or ray-tracing and hidden curve removal. In this paper we present an algorithm for decomposing a given surface into regions so that each region is either completely visible or hidden from a given viewpoint. Initially, it decomposes the domain of each surface based on silhouettes and boundary curves. To compute the exact visibility, we introduce a notion of visibility curves obtained by projection of silhouette and boundary curves and decomposition of the surface into nonoverlapping regions. These curves are computed using marching methods and we present techniques to compute all the components. The nonoverlapping and visible portions of the surface are represented as trimmed surfaces and we present a representation based on polygon trapezoidation algorithms. The algorithms presented use some recently developed algorithms from computational geometry like triangulation of simple polygons and point location. Given the nonoverlapping regions, we use an existing randomized algorithm for visibility computation. We also present results from a preliminary implementation of our algorithm.     

Multiterminal source coding for multiview images under wireless fading channels

This paper addresses the problem of wireless transmission of a captured scene from multiple cameras, which do not communicate among each other, to a joint decoder. Correlation among different camera views calls for distributed source coding for efficient multiview image compression. The fact that cameras are placed within a short range of each other results in a high level of interference, multipath fading, and noise effects during communications. We develop a novel two-camera system, that employs multiterminal source coding and complete complementary data spreading, so that while the former technique exploits the statistical correlation between camera views, and performs joint compression to reduce transmission rates, the spreading technique will protect transmitted data by mitigating the effects of wireless fading channels. Our results indicate that the proposed system is competitive when compared to two independently JPEG encoded streams at low to medium transmission rates.     

Orienting raw point sets by global contraction and visibility voting

We present a global method for consistently orienting a defective raw point set with noise, non-uniformities and thin sharp features. Our method seamlessly combines two simple but effective techniques—constrained Laplacian smoothing and visibility voting—to tackle this challenge. First, we apply a Laplacian contraction to the given point cloud, which shrinks the shape a little bit. Each shrunk point corresponds to an input point and shares a visibility confidence assigned by voting from multiple viewpoints. The confidence is increased (resp. decreased) if the input point (resp. its corresponding shrunk point) is visible. Then, the initial normals estimated by principal component analysis are flipped according to the contraction vectors from shrunk points to the corresponding input points and the visibility confidence. Finally, we apply a Laplacian smoothing twice to correct the orientation of points with zero or low confidence. Our method is conceptually simple and easy to implement, without resorting to any complicated data structures and advanced solvers. Numerous experiments demonstrate that our method can orient the defective raw point clouds in a consistent manner. By taking advantage of our orientation information, the classical implicit surface reconstruction algorithms can faithfully generate the surface.     

Two-view multibody structure-and-motion with outliers through model selection

Multibody structure-and-motion (MSaM) is the problem in establishing the multiple-view geometry of several views of a 3D scene taken at different times, where the scene consists of multiple rigid objects moving relative to each other. We examine the case of two views. The setting is the following: Given are a set of corresponding image points in two images, which originate from an unknown number of moving scene objects, each giving rise to a motion model. Furthermore, the measurement noise is unknown, and there are a number of gross errors, which are outliers to all models. The task is to find an optimal set of motion models for the measurements. It is solved through Monte-Carlo sampling, careful statistical analysis of the sampled set of motion models, and simultaneous selection of multiple motion models to best explain the measurements. The framework is not restricted to any particular model selection mechanism because it is developed from a Bayesian viewpoint: different model selection criteria are seen as different priors for the set of moving objects, which allow one to bias the selection procedure for different purposes.     

多视三维重构中连续能量模型的凸优化

利用融合了轮廓线及体视的序列图像信息,提出了一个面向多视三维重构的稳健能量模型;为了适配于可视性约束,提出一种针对该能量模型的连续全局优化方法;为了保证栅格连通性选择的一致性及独立性,实施了全局连续优化的超松弛离散化。实例证明,该方法的实用性好,极大地减少了算法处理的内存开销,实现了在更高分辨率上有效的多视重构。     

Multiview segmentation and tracking of dynamic occluding layers

We present an algorithm for the layered segmentation of video data in multiple views. The approach is based on computing the parameters of a layered representation of the scene in which each layer is modelled by its motion, appearance and occupancy, where occupancy describes, probabilistically, the layer’s spatial extent and not simply its segmentation in a particular view. The problem is formulated as the MAP estimation of all layer parameters conditioned on those at the previous time step; i.e., a sequential estimation problem that is equivalent to tracking multiple objects in a given number views. Expectation–Maximisation is used to establish layer posterior probabilities for both occupancy and visibility, which are represented distinctly. Evidence from areas in each view which are described poorly under the model is used to propose new layers automatically. Since these potential new layers often occur at the fringes of images, the algorithm is able to segment and track these in a single view until such time as a suitable candidate match is discovered in the other views. The algorithm is shown to be very effective at segmenting and tracking non-rigid objects and can cope with extreme occlusion. We demonstrate an application of this representation to dynamic novel view synthesis.     

Conservative Visibility and Strong Occlusion for Viewspace Partitioning of Densely Occluded Scenes

Computing the visibility of out-door scenes is often much harder than of in-door scenes. A typical urban scene, for example, is densely occluded, and it is effective to precompute its visibility space, since from a given point only a small fraction of the scene is visible. The difficulty is that although the majority of objects are hidden, some parts might be visible at a distance in an arbitrary location, and it is not clear how to detect them quickly. In this paper we present a method to partition the viewspace into cells containing a conservative superset of the visible objects. For a given cell the method tests the visibility of all the objects in the scene. For each object it searches for a strong occluder which guarantees that the object is not visible from any point within the cell. We show analytically that in a densely occluded scene, the vast majority of objects are strongly occluded, and the overhead of using conservative visibility (rather than visibility) is small. These results are further supported by our experimental results. We also analyze the cost of the method and discuss its effectiveness.     

Fourier domain representation of planar curves for recognition in multiple views

Recognition of planar shapes is an important problem in computer vision and pattern recognition. The same planar object contour imaged from different cameras or from different viewpoints looks different and their recognition is non-trivial. Traditional shape recognition deals with views of the shapes that differ only by simple rotations, translations, and scaling. However, shapes suffer more serious deformation between two general views and hence recognition approaches designed to handle translations, rotations, and/or scaling would prove to be insufficient. Many algebraic relations between matching primitives in multiple views have been identified recently. In this paper, we explore how shape properties and multiview relations can be combined to recognize planar shapes across multiple views. We propose novel recognition constraints that a planar shape boundary must satisfy in multiple views. The constraints are on the rank of a Fourier-domain measurement matrix computed from the points on the shape boundary. Our method can additionally compute the correspondence between the curve points after a match is established. We demonstrate the applications of these constraints experimentally on a number of synthetic and real images.     

Fluid in Video: Augmenting Real Video with Simulated Fluids

We present a technique for coupling simulated fluid phenomena that interact with real dynamic scenes captured as a binocular video sequence. We first process the binocular video sequence to obtain a complete 3D reconstruction of the scene, including velocity information. We use stereo for the visible parts of 3D geometry and surface completion to fill the missing regions. We then perform fluid simulation within a 3D domain that contains the object, enabling one‐way coupling from the video to the fluid. In order to maintain temporal consistency of the reconstructed scene and the animated fluid across frames, we develop a geometry tracking algorithm that combines optic flow and depth information with a novel technique for “velocity completion”. The velocity completion technique uses local rigidity constraints to hypothesize a motion field for the entire 3D shape, which is then used to propagate and filter the reconstructed shape over time. This approach not only generates smoothly varying geometry across time, but also simultaneously provides the necessary boundary conditions for one‐way coupling between the dynamic geometry and the simulated fluid. Finally, we employ a GPU based scheme for rendering the synthetic fluid in the real video, taking refraction and scene texture into account.