Building Roadmaps of Minima and Transitions in Visual Models

Becoming trapped in suboptimal local minima is a perennial problem when optimizing visual models, particularly in applications like monocular human body tracking where complicated parametric models are repeatedly fitted to ambiguous image measurements. We show that trapping can be significantly reduced by building roadmaps of nearby minima linked by transition pathways—paths leading over low mountain passes in the cost surface—found by locating the transition state (codimension-1 saddle point) at the top of the pass and then sliding downhill to the next minimum. We present two families of transition-state-finding algorithms based on local optimization. In eigenvector tracking, unconstrained Newton minimization is modified to climb uphill towards a transition state, while in hypersurface sweeping, a moving hypersurface is swept through the space and moving local minima within it are tracked using a constrained Newton method. These widely applicable numerical methods, which appear not to be known in vision and optimization, generalize methods from computational chemistry where finding transition states is critical for predicting reaction parameters. Experiments on the challenging problem of estimating 3D human pose from monocular images show that our algorithms find nearby transition states and minima very efficiently, but also underline the disturbingly large numbers of minima that can exist in this and similar model based vision problems.     

Feedback Flow Control with Hysteresial Techniques for Multimedia Retrievals

The objective of this paper is to demonstrate the advantages of using hysteresial techniques in flow control mechanisms. The work scenario consists of a server transmitting video information to a group of clients. Each client stores the information in a buffer and then plays it back at a set consumption rate. To avoid overflow or underflow in the buffer, the client sends control messages (feedbacks) to the server which order adjustments in the transmission rate. Hysteresial techniques allow the minimization of this signaling traffic, even when there are rapid fluctuations in the buffer occupation. As a first step, an analytical model is developed using Markovian processes. This produces expressions of the most relevant parameters and allows the evaluation of the proposed flow control mechanism. The Markovian model has been compared with the performance of a multimedia application for video distribution, running in a real scenario. The results show that the model represents qualitatively the real scenario and consequently validates the model usefulness.     

3-D Reconstruction of Shaded Objects from Multiple Images Under Unknown Illumination

We propose a variational algorithm to jointly estimate the shape, albedo, and light configuration of a Lambertian scene from a collection of images taken from different vantage points. Our work can be thought of as extending classical multi-view stereo to cases where point correspondence cannot be established, or extending classical shape from shading to the case of multiple views with unknown light sources. We show that a first naive formalization of this problem yields algorithms that are numerically unstable, no matter how close the initialization is to the true geometry. We then propose a computational scheme to overcome this problem, resulting in provably stable algorithms that converge to (local) minima of the cost functional. We develop a new model that explicitly enforces positivity in the light sources with the assumption that the object is Lambertian and its albedo is piecewise constant and show that the new model significantly improves the accuracy and robustness relative to existing approaches.     

Incremental View Maintenance By Base Relation Tagging in Distributed Databases

The incremental view maintenance problem deals with the efficient updating of materialized views in response to updates to base relations. This paper considers the problem in a distributed database environment, with communication cost minimization as the primary objective. The views considered are defined based on the relational join operation. The approach is to use yes/no tags as auxiliary data on tuples in the base relations to indicate whether the tuples participate in joins. These tags will help avoid sending irrelevant data over the network and thus reduce the communication cost. Two basic view maintenance algorithms are proposed using the tags. In addition to reducing communication costs, an important feature of these two basic algorithms is that they derive the exact change to views without looking at the old views. This feature allows us to maintain certain aggregates on views without actually materializing the views themselves; this feature is useful in applications such as active databases where many conditions or constraints must be tested whenever updates occur, since a condition is true exactly when some corresponding view has nonzero number of tuples. The paper then combines the use of tags with the counting algorithm to derive a tagged counting algorithm that further reduces the communication cost. The paper illustrates the algorithms by examples and studies their performance via a statistical analysis. The illustrating examples and the performance analysis show that, under uniform distribution with reasonable join participation rates, the use of tags significantly improves the efficiency of view maintenance over similar algorithms without tags. The performance analysis also identifies the situations where a particular algorithm is superior to others. The use of tags for memoing values of subexpressions in a view definition is also explored in the paper.     

Quantum annealing for combinatorial clustering

Clustering is a powerful machine learning technique that groups “similar” data points based on their characteristics. Many clustering algorithms work by approximating the minimization of an objective function, namely the sum of within-the-cluster distances between points. The straightforward approach involves examining all the possible assignments of points to each of the clusters. This approach guarantees the solution will be a global minimum; however, the number of possible assignments scales quickly with the number of data points and becomes computationally intractable even for very small datasets. In order to circumvent this issue, cost function minima are found using popular local search-based heuristic approaches such as k-means and hierarchical clustering. Due to their greedy nature, such techniques do not guarantee that a global minimum will be found and can lead to sub-optimal clustering assignments. Other classes of global search-based techniques, such as simulated annealing, tabu search, and genetic algorithms, may offer better quality results but can be too time-consuming to implement. In this work, we describe how quantum annealing can be used to carry out clustering. We map the clustering objective to a quadratic binary optimization problem and discuss two clustering algorithms which are then implemented on commercially available quantum annealing hardware, as well as on a purely classical solver “qbsolv.” The first algorithm assigns N data points to K clusters, and the second one can be used to perform binary clustering in a hierarchical manner. We present our results in the form of benchmarks against well-known k-means clustering and discuss the advantages and disadvantages of the proposed techniques.     

Image Registration, Optical Flow and Local Rigidity

We address the theoretical problems of optical flow estimation and image registration in a multi-scale framework in any dimension. Much work has been done based on the minimization of a distance between a first image and a second image after applying deformation or motion field. Usually no justification is given about convergence of the algorithm used. We start by showing, in the translation case, that convergence to the global minimum is made easier by applying a low pass filter to the images hence making the energy convex enough. In order to keep convergence to the global minimum in the general case, we introduce a local rigidity hypothesis on the unknown deformation. We then deduce a new natural motion constraint equation (MCE) at each scale using the Dirichlet low pass operator. This transforms the problem to solving the energy minimization in a finite dimensional subspace of approximation obtained through Fourier Decomposition. This allows us to derive sufficient conditions for convergence of a new multi-scale and iterative motion estimation/registration scheme towards a global minimum of the usual nonlinear energy instead of a local minimum as did all previous methods. Although some of the sufficient conditions cannot always be fulfilled because of the absence of the necessary a priori knowledge on the motion, we use an implicit approach. We illustrate our method by showing results on synthetic and real examples in dimension 1 (signal matching, Stereo) and 2 (Motion, Registration, Morphing), including large deformation experiments.     

New Linear Program Performance Bounds for Closed Queueing Networks

We develop new linear program performance bounds for closed reentrantqueueing networks based on an inequality relaxation of the averagecost equation. The approach exploits the fact that the transitionprobabilities under certain policies of closed queueing networksare invariant within certain regions of the state space. Thisinvariance suggests the use of a piecewise quadratic functionas a surrogate for the differential cost function. The linearprogramming throughput bounds obtained are provably tighter thanpreviously known bounds at the cost of increased computationalcomplexity. Functional throughput bounds parameterized by thefixed customer population N are obtained, alongwith a bound on the limiting throughput as N + .We show that one may obtain reduced complexity bounds while stillretaining superiority.     

Optimality tests for fixed points of the fuzzy c-means algorithm

The Fuzzy c-Means(FCM) clustering algorithms are known to converge to either local minima or saddle points of the objective function which defines the FCM method. The object of this paper is to derive efficient numerical tests for local extrema of the FCM functional that enable one to identify each candidate as a local minimum or saddle point. Numerical examples of the theory derived illustrate that the tests proposed cover all possible cases.     

A new gradient based particle swarm optimization algorithm for accurate computation of global minimum

Stochastic optimization algorithms like genetic algorithms (GAs) and particle swarm optimization (PSO) algorithms perform global optimization but waste computational effort by doing a random search. On the other hand deterministic algorithms like gradient descent converge rapidly but may get stuck in local minima of multimodal functions. Thus, an approach that combines the strengths of stochastic and deterministic optimization schemes but avoids their weaknesses is of interest. This paper presents a new hybrid optimization algorithm that combines the PSO algorithm and gradient-based local search algorithms to achieve faster convergence and better accuracy of final solution without getting trapped in local minima. In the new gradient-based PSO algorithm, referred to as the GPSO algorithm, the PSO algorithm is used for global exploration and a gradient based scheme is used for accurate local exploration. The global minimum is located by a process of finding progressively better local minima. The GPSO algorithm avoids the use of inertial weights and constriction coefficients which can cause the PSO algorithm to converge to a local minimum if improperly chosen. The De Jong test suite of benchmark optimization problems was used to test the new algorithm and facilitate comparison with the classical PSO algorithm. The GPSO algorithm is compared to four different refinements of the PSO algorithm from the literature and shown to converge faster to a significantly more accurate final solution for a variety of benchmark test functions.     

一种快速逃离局部极小点的BP算法

针对反向传播(BP)算法容易陷入局部极小点的问题,提出了一种改进价值函数,使其快速收敛到全局最小点的方法。对扩展的异或问题正弦函数模拟进行了仿真实验,结果对比表明,改进的BP算法能快速逃离局部极小点,收敛到全局最小点,达到了期望的效果。     

Simulated annealing and weight decay in adaptive learning: theSARPROP algorithm

A problem with gradient descent algorithms is that they can converge to poorly performing local minima. Global optimization algorithms address this problem, but at the cost of greatly increased training times. This work examines combining gradient descent with the global optimization technique of simulated annealing (SA). Simulated annealing in the form of noise and weight decay is added to resiliant backpropagation (RPROP), a powerful gradient descent algorithm for training feedforward neural networks. The resulting algorithm, SARPROP, is shown through various simulations not only to be able to escape local minima, but is also able to maintain, and often improve the training times of the RPROP algorithm. In addition, SARPROP may be used with a restart training phase which allows a more thorough search of the error surface and provides an automatic annealing schedule.     

Some PAC-Bayesian Theorems

This paper gives PAC guarantees for Bayesian algorithms—algorithms that optimize risk minimization expressions involving a prior probability and a likelihood for the training data. PAC-Bayesian algorithms are motivated by a desire to provide an informative prior encoding information about the expected experimental setting but still having PAC performance guarantees over all IID settings. The PAC-Bayesian theorems given here apply to an arbitrary prior measure on an arbitrary concept space. These theorems provide an alternative to the use of VC dimension in proving PAC bounds for parameterized concepts.     

Economical algorithms for the interpretation of the algorithmic language INCOL

Conclusions The memory gain given by the present algorithms, in comparison with the algorithms of [2], depends on the class of problems to be solved and can be quite substantial in the solution of problems in which the unknowns have long forms as their values. The cost in time connected with the more complex structures of the lists must, in general, be compensated by the less frequent calls to the procedures described in [2] for the so-called garbage-collector-optimizer and by the replacement of copying of values by the copying of references (the marks ). It remains to be noted that if the values of the unknowns or the function identifiers are constants, then the algorithms reduce to those described in [2].Translated from Kibernetika, No. 1, pp. 69–74, January–February, 1977.     

Numerical computation of H∞ optimal performance

We present new algorithms for computing theH optimal performance for a class of single-input/single-output (SISO) infinite-dimensional systems. The algorithms here only require use of one or two fast Fourier transforms (FFT) and Cholesky decompositions; hence the algorithms are particularly simple and easy to implement. Numerical examples show that the algorithms are stable and efficient and converge rapidly. The method has wide applications including to theH optimal control of distributed parameter systems. We illustrate the technique with applications to some delay problems and a partial differential equation (PDE) model. The algorithms we present are also an attractive approach to the solution of high-order finite-dimensional models for which use of state space methods would present computational difficulties.     

General Convergence Results for Linear Discriminant Updates

The problem of learning linear-discriminant concepts can be solved by various mistake-driven update procedures, including the Winnow family of algorithms and the well-known Perceptron algorithm. In this paper we define the general class of quasi-additive algorithms, which includes Perceptron and Winnow as special cases. We give a single proof of convergence that covers a broad subset of algorithms in this class, including both Perceptron and Winnow, but also many new algorithms. Our proof hinges on analyzing a generic measure of progress construction that gives insight as to when and how such algorithms converge.Our measure of progress construction also permits us to obtain good mistake bounds for individual algorithms. We apply our unified analysis to new algorithms as well as existing algorithms. When applied to known algorithms, our method automatically produces close variants of existing proofs (recovering similar bounds)—thus showing that, in a certain sense, these seemingly diverse results are fundamentally isomorphic. However, we also demonstrate that the unifying principles are more broadly applicable, and analyze a new class of algorithms that smoothly interpolate between the additive-update behavior of Perceptron and the multiplicative-update behavior of Winnow.     

基于预设时间收敛的分布式优化算法

考虑一类分布式优化问题,其目标是通过局部信息交互,使得局部成本函数之和构成的全局成本函数最小.针对该类问题,通过引入时基发生器(TBG),提出两种基于预设时间收敛的分布式比例积分(PI)优化算法.与现有的基于有限/固定时间收敛的分布式优化算法相比,所提出算法的收敛时间不依赖于系统的初值和参数,且可以任意预先设计.此外,在全局成本函数关于最优值点有限强凸,局部成本函数为可微的凸函数,且具有局部Lipschitz梯度的条件下,通过Lyapunov理论证明了所提算法都能实现预设时间收敛.最后,通过数值仿真验证了所提出算法的有效性.     

Verifying Global Minima for L 2 Minimization Problems in Multiple View Geometry

We consider the least-squares (L2) minimization problems in multiple view geometry for triangulation, homography, camera resectioning and structure-and-motion with known rotation, or known plane. Although optimal algorithms have been given for these problems under an L-infinity cost function, finding optimal least-squares solutions to these problems is difficult, since the cost functions are not convex, and in the worst case may have multiple minima. Iterative methods can be used to find a good solution, but this may be a local minimum. This paper provides a method for verifying whether a local-minimum solution is globally optimal, by providing a simple and rapid test involving the Hessian of the cost function. The basic idea is that by showing that the cost function is convex in a restricted but large enough neighbourhood, a sufficient condition for global optimality is obtained. The method is tested on numerous problem instances of real data sets. In the vast majority of cases we are able to verify that the solutions are optimal, in particular, for small to medium-scale problems.     

Preconditioning index set transformations for time-optimal affine scheduling

Many regular algorithms, suitable for VLSI implementation, are naturally described by sets of integer index vectors together with a rule that assigns a computation to each vector. Regular VLSI structures for such algorithms can be found by mapping the index vectors to a discrete space-time with integer coordinates. If the scope is restricted to linear or affine mappings, then the minimization of the execution time for the VLSI implementation with respect to the space-time mapping is essentially an integer linear programming (ILP) problem. If the entries in the vector describing the time function must be integers, ILP techniques can be applied directly. There are, however, index sets that allow space-time mappings with rational, nonintegral entries. In such cases, ILP will not consider all possible affine time functions and an optimal solution may go unnoticed. In this paper we give sufficient conditions on the index set for when only integer time functions are allowed. We also give a general algorithm to find a preconditioning affine transformation of the index set, such that the transformed index set allows only integer time functions. ILP methods can then be used to find time-optimal architectures for the transformed algorithm. This considerably extends the class of algorithms for which time-optimal VLSI structures can be found.Communicated by M. Snir.     

CCCP algorithms to minimize the Bethe and Kikuchi free energies: convergent alternatives to belief propagation

This article introduces a class of discrete iterative algorithms that are provably convergent alternatives to belief propagation (BP) and generalized belief propagation (GBP). Our work builds on recent results by Yedidia, Freeman, and Weiss (2000), who showed that the fixed points of BP and GBP algorithms correspond to extrema of the Bethe and Kikuchi free energies, respectively. We obtain two algorithms by applying CCCP to the Bethe and Kikuchi free energies, respectively (CCCP is a procedure, introduced here, for obtaining discrete iterative algorithms by decomposing a cost function into a concave and a convex part). We implement our CCCP algorithms on two- and three-dimensional spin glasses and compare their results to BP and GBP. Our simulations show that the CCCP algorithms are stable and converge very quickly (the speed of CCCP is similar to that of BP and GBP). Unlike CCCP, BP will often not converge for these problems (GBP usually, but not always, converges). The results found by CCCP applied to the Bethe or Kikuchi free energies are equivalent, or slightly better than, those found by BP or GBP, respectively (when BP and GBP converge). Note that for these, and other problems, BP and GBP give very accurate results (see Yedidia et al., 2000), and failure to converge is their major error mode. Finally, we point out that our algorithms have a large range of inference and learning applications.