Bayesian Scheme for Interactive Colourization,Recolourization and Image/Video Editing

We propose a general image and video editing method based on a Bayesian segmentation framework. In the first stage, classes are established from scribbles made by a user on the image. These scribbles can be considered as a multi‐map (multi‐label map) that defines the boundary conditions of a probability measure field to be computed for each pixel. In the second stage, the global minima of a positive definite quadratic cost function with linear constraints, is calculated to find the probability measure field. The components of such a probability measure field express the degree of each pixel belonging to spatially smooth classes. Finally, the computed probabilities (memberships) are used for defining the weights of a linear combination of user provided colours or effects associated to each class. The proposed method allows the application of different operators, selected interactively by the user, over part or the whole image without needing to recompute the memberships. We present applications to colourization, recolourization, editing and photomontage tasks.     

Interactive Editing of Lighting and Materials using a Bivariate BRDF Representation

We present a new Precomputed Radiance Transfer (PRT) algorithm based on a two dimensional representation of isotropic BRDFs. Our approach involves precomputing matrices that allow quickly mapping environment lighting, which is represented in the global coordinate system, and the surface BRDFs, which are represented in a bivariate domain, to the local hemisphere at a surface location where the reflection integral is evaluated. When the lighting and BRDFs are represented in a wavelet basis, these rotation matrices are sparse and can be efficiently stored and combined with pre‐computed visibility at run‐time. Compared to prior techniques that also precompute wavelet rotation matrices, our method allows full control over the lighting and materials due to the way the BRDF is represented. Furthermore, this bivariate parameterization preserves sharp specular peaks and grazing effects that are attenuated in conventional parameterizations. We demonstrate a prototype rendering system that achieves real‐time framerates while lighting and materials are edited.     

Click Carving: Interactive Object Segmentation in Images and Videos with Point Clicks

International Journal of Computer Vision - We present a novel form of interactive object segmentation called Click Carving which enables accurate segmentation of objects in images and videos with...     

Interactive Pattern Recognition: A System and Data Structure

In this paper a general-purpose interactive pattern recognition system is built using limited resources. The system utilizes Ribak's [5] relationship matrix (R-matrix) as a data structure and a PDP-9 minicomputer as hardware. The software developed for the interactive system along with the implemented idea of "vague" relations gives the human operator flexibility and aid in interactive file creation. Files created to describe complex patterns can be formed by merging simpler subfiles of different portions of that pattem. In addition to merging files (union operation), common elements between files can be detected using an intersection operation. User-defined routines are readily interfaced to the interactive system, thus increasing the system flexibility and giving the human operator a tailor-made effect.     

Mediated Perceptions: Contributions of Phenomenological Film Theory to Understanding the Interactive Video Experience

This article examines the spatial relationships between avatars (i.e., graphical identities or icons) over time in a 2-dimensional online chat environment. The Spatial Distance Analysis Program (SDAP) was developed to measure the distance between avatars in a specially designed Palace environment. Correlations between distance and interpersonal communication constructs of (1) conversational appropriateness, (2) social attraction, and (3) uncertainty reduction indicate that distance effects are significant in an online environment. Specifically, it was found that general conversational appropriateness mediated between uncertainty reduction and specific conversational appropriateness for individuals who moved closer together and farther apart over time, respectively. Furthermore, the relationship between social attraction and distance indicated a significant positive parabolic function; that social attraction (i.e., liking) decreased at middle distances and increased at low and high distances. This finding suggests that there are three interpersonal distance zones in online communication.     

Watch to Edit: Video Retargeting using Gaze

We present a novel approach to optimally retarget videos for varied displays with differing aspect ratios by preserving salient scene content discovered via eye tracking. Our algorithm performs editing with cut, pan and zoom operations by optimizing the path of a cropping window within the original video while seeking to (i) preserve salient regions, and (ii) adhere to the principles of cinematography. Our approach is (a) content agnostic as the same methodology is employed to re‐edit a wide‐angle video recording or a close‐up movie sequence captured with a static or moving camera, and (b) independent of video length and can in principle re‐edit an entire movie in one shot. Our algorithm consists of two steps. The first step employs gaze transition cues to detect time stamps where new cuts are to be introduced in the original video via dynamic programming. A subsequent step optimizes the cropping window path (to create pan and zoom effects), while accounting for the original and new cuts. The cropping window path is designed to include maximum gaze information, and is composed of piecewise constant, linear and parabolic segments. It is obtained via L(1) regularized convex optimization which ensures a smooth viewing experience. We test our approach on a wide variety of videos and demonstrate significant improvement over the state‐of‐the‐art, both in terms of computational complexity and qualitative aspects. A study performed with 16 users confirms that our approach results in a superior viewing experience as compared to gaze driven re‐editing [ JSSH15 ] and letterboxing methods, especially for wide‐angle static camera recordings.     

Convertibility of Function Points into COSMIC Function Points: A study using Piecewise Linear Regression

Background

COSMIC Function Points and traditional Function Points (i.e., IFPUG Function Points and more recent variation of Function Points, such as NESMA and FISMA) are probably the best known and most widely used Functional Size Measurement methods. The relationship between the two kinds of Function Points still needs to be investigated. If traditional Function Points could be accurately converted into COSMIC Function Points and vice versa, then, by measuring one kind of Function Points, one would be able to obtain the other kind of Function Points, and one might measure one or the other kind interchangeably. Several studies have been performed to evaluate whether a correlation or a conversion function between the two measures exists. Specifically, it has been suggested that the relationship between traditional Function Points and COSMIC Function Points may not be linear, i.e., the value of COSMIC Function Points seems to increase more than proportionally to an increase of traditional Function Points.

Objective

This paper aims at verifying this hypothesis using available datasets that collect both FP and CFP size measures.

Method

Rigorous statistical analysis techniques are used, specifically Piecewise Linear Regression, whose applicability conditions are systematically checked. The Piecewise Linear Regression curve is a series of interconnected segments. In this paper, we focused on Piecewise Linear Regression curves composed of two segments. We also used Linear and Parabolic Regressions, to check if and to what extent Piecewise Linear Regression may provide an advantage over other regression techniques. We used two categories of regression techniques: Ordinary Least Squares regression is based on the usual minimization of the sum of squares of the residuals, or, equivalently, on the minimization of the average squared residual; Least Median of Squares regression is a robust regression technique that is based on the minimization of the median squared residual. Using a robust regression technique helps filter out the excessive influence of outliers.

Results

It appears that the analysis of the relationship between traditional Function Points and COSMIC Function Points based on the aforementioned data analysis techniques yields valid significant models. However, different results for the various available datasets are achieved. In practice, we obtained statistically valid linear, piecewise linear, and non-linear conversion formulas for several datasets. In general, none of these is better than the others in a statistically significant manner.

Conclusions

Practitioners interested in the conversion of FP measures into CFP (or vice versa) cannot just pick a conversion model and be sure that it will yield the best results. All the regression models we tested provide good results with some datasets. In practice, all the models described in the paper - in particular, both linear and non-linear ones - should be evaluated in order to identify the ones that are best suited for the specific dataset at hand.     

Video compression using MPEG 7-MBBMC: techniques for communication

Communication is termed as exchanging the information (audio, video, text and image) from one end (transmitter) to another end (receiver). When video data are compressed and transmitted to another side, compression reduces the bandwidth size and memory required to transmit the video. Some traditional techniques are used in video transmission but it includes drawbacks, such as more compression time and low quality due to compression. To overcome these drawbacks the MPEG7-MBBMC (Modified Block Based Motion Compensated) technique is developed. Here the input video signals are collected from the dataset and the signals are splitted into three bands. Discrete Wavelet Transform (DWT) is applied for each band and quantization process occurs. The DWT and quantization process are applied in the MPEG7 compression, which offers high compression factors. Next, encoder is applied to convert the packets into small packets by using modified block based motion compensated (MBBMC) technique. The Motion compensation establishes a correspondence between elements of nearby images in the video sequence. The Forward Error Correction (FEC) is used to reduce the distortion in the encoder video packet. Then the Channel Pattern Integration (CPI) is applied to find the best available channel. The encoded video packets are transmitted by the best available channel. In receiver side the error correction code is applied to decode the video packets and reconstructs the decoded packet by decompression. It improves the quality of the video and in future it will help for much development in the field of multimedia.

     

PVP: Personalized Video Prior for Editable Dynamic Portraits using StyleGAN

Portrait synthesis creates realistic digital avatars which enable users to interact with others in a compelling way. Recent advances in StyleGAN and its extensions have shown promising results in synthesizing photorealistic and accurate reconstruction of human faces. However, previous methods often focus on frontal face synthesis and most methods are not able to handle large head rotations due to the training data distribution of StyleGAN. In this work, our goal is to take as input a monocular video of a face, and create an editable dynamic portrait able to handle extreme head poses. The user can create novel viewpoints, edit the appearance, and animate the face. Our method utilizes pivotal tuning inversion (PTI) to learn a personalized video prior from a monocular video sequence. Then we can input pose and expression coefficients to MLPs and manipulate the latent vectors to synthesize different viewpoints and expressions of the subject. We also propose novel loss functions to further disentangle pose and expression in the latent space. Our algorithm shows much better performance over previous approaches on monocular video datasets, and it is also capable of running in real-time at 54 FPS on an RTX 3080.     

Video collage: presenting a video sequence using a single image

The explosive growth of video data demands the video presentation technique which supports fast browsing of video content. In this paper, we present an automatic procedure for constructing a compact synthesized collage from a video sequence. The synthesized image, called “Video Collage”, is a kind of static video summary—to select the most representative images from video, to extract salient regions of interest (ROIs) from these images, and to seamlessly arrange ROIs on a given canvas with the temporal structure of video content preserved. We formulate the generation of Video Collage as a unified energy minimization problem in which each of above desirability is represented by an energy term. We start from the basic setting of Video Collage in which both the shape of ROIs and collage are fixed as rectangular, and then show how it can support arbitrary shapes of ROIs, as well as a variety of collage templates and region of interest (ROI) arrangement layouts (i.e., book, diagonal, and spiral). The experiments show its effectiveness to present a video in a very compact and visually appealing form while preserving the necessary information to understand the video.

Interactive lectures: Effective teaching and learning in lectures using wireless networks

Among the severe instructional shortcomings of lectures is their lack of interactivity. To overcome these problems, the learning environment of the interactive lecture is introduced, in which students and lecturer interact by using mobile computers in a wireless network. In an experimental study, 44 students participated in interactive and in conventional lecture meetings. A quiz service that allowed question to be posted, students’ answers to be evaluated, and the results to be presented graphically, was implemented on mobile computers. The participants rated the interactive condition better, also reporting higher levels of attention, activity and perceived learning success for this condition. Objective measures indicated a slight but insignificant difference toward better learning results in the interactive condition.     

IFTrace: Video segmentation of deformable objects using the Image Foresting Transform

We introduce IFTrace, a method for video segmentation of deformable objects. The algorithm makes minimal assumptions about the nature of the tracked object: basically, that it consists of a few connected regions, and has a well-defined border. The objects to be tracked are interactively segmented in the first frame of the video, and a set of markers is then automatically selected in the interior and immediate surroundings of the object. These markers are then located in the next frame by a combination of KLT feature finding and motion extrapolation. Object boundaries are then identified from these markers by the Image Foresting Transform (IFT). These steps are repeated for all subsequent frames until the end of the movie. Thanks to the IFT and a special boundary detection operator, IFTrace can reliably track deformable objects in the presence of partial and total occlusions, camera motion, lighting and color changes, and other complications. Tests on real videos show that the IFT is better suited to this task than Graph-Cut methods, and that IFTrace is more robust than other state-of-the art algorithms – namely, the OpenCV Snake and CamShift algorithms, Hess’s Particle-Filter, and Zhong and Chang’s method based on spatio-temporal consistency.     

Interactive tools for conceptual programming using APL-graphics: The integral and derivative operators

The notion of conceptual programming is defined by way of illustrations in the context of interactive problem solving. The numerical implementations of a DERIVATIVE and an INTEGRAL operators are discussed. These operators have been integrated, together with some basic graphical operators, into the APL system in a manner consistent with the syntax of the host language, providing extremely powerful tools for interactive problem solving. The performance of the system as well as some error analysis are demonstrated.     

Time Lattice: A Data Structure for the Interactive Visual Analysis of Large Time Series

Advances in technology coupled with the availability of low‐cost sensors have resulted in the continuous generation of large time series from several sources. In order to visually explore and compare these time series at different scales, analysts need to execute online analytical processing (OLAP) queries that include constraints and group‐by's at multiple temporal hierarchies. Effective visual analysis requires these queries to be interactive. However, while existing OLAP cube‐based structures can support interactive query rates, the exponential memory requirement to materialize the data cube is often unsuitable for large data sets. Moreover, none of the recent space‐efficient cube data structures allow for updates. Thus, the cube must be re‐computed whenever there is new data, making them impractical in a streaming scenario. We propose Time Lattice, a memory‐efficient data structure that makes use of the implicit temporal hierarchy to enable interactive OLAP queries over large time series. Time Lattice is a subset of a fully materialized cube and is designed to handle fast updates and streaming data. We perform an experimental evaluation which shows that the space efficiency of the data structure does not hamper its performance when compared to the state of the art. In collaboration with signal processing and acoustics research scientists, we use the Time Lattice data structure to design the Noise Profiler, a web‐based visualization framework that supports the analysis of noise from cities. We demonstrate the utility of Noise Profiler through a set of case studies.     

Correction to: Video sequence feature extraction and segmentation using likelihood regression model

Multimedia Tools and Applications -     

DocuBurst: Visualizing Document Content using Language Structure

Textual data is at the forefront of information management problems today. One response has been the development of visualizations of text data. These visualizations, commonly based on simple attributes such as relative word frequency, have become increasingly popular tools. We extend this direction, presenting the first visualization of document content which combines word frequency with the human-created structure in lexical databases to create a visualization that also reflects semantic content. DocuBurst is a radial, space-filling layout of hyponymy (the IS-A relation), overlaid with occurrence counts of words in a document of interest to provide visual summaries at varying levels of granularity. Interactive document analysis is supported with geometric and semantic zoom, selectable focus on individual words, and linked access to source text.     

Video telephony - quality of experience: a simple QoE model to assess video calls using subjective approach

Multimedia Tools and Applications - One problem about using video calls or video telephony applications is a quality issue. Therefore, this paper contributes the simple QoE model, a mathematical...     

Interactive computer-aided control design using quantitative feedback theory: the problem of vertical movement stabilization on a high-speed ferry

In a first approximation, the vertical acceleration associated with pitch motion can be considered as the main cause of motion sickness, which is without a doubt one of the most unpleasant disadvantages of maritime transport. The reduction of motion sickness can be stated as a monovariable regulation problem of a highly perturbed system. This work presents the design of a monovariable robust controller with quantitative feedback theory (QFT) for reducing the vertical movement on a high-speed ferry. The different stages of QFT methodology have been done with the help of the software tool QFTIT (Quantitative Feedback Theory Interactive Tool). This is a free software tool that is characterized by its ease of use and interactive nature. The designed regulator is validated experimentally in sea behaviour trials with a scaled down replica 1/25 the size of a high-speed ferry. The designed regulator is also compared with a gain-scheduling scheme using a proportional and derivative controller (PD).