Mode Decomposition Evolution Equations

Partial differential equation (PDE) based methods have become some of the most powerful tools for exploring the fundamental problems in signal processing, image processing, computer vision, machine vision and artificial intelligence in the past two decades. The advantages of PDE based approaches are that they can be made fully automatic, robust for the analysis of images, videos and high dimensional data. A fundamental question is whether one can use PDEs to perform all the basic tasks in the image processing. If one can devise PDEs to perform full-scale mode decomposition for signals and images, the modes thus generated would be very useful for secondary processing to meet the needs in various types of signal and image processing. Despite of great progress in PDE based image analysis in the past two decades, the basic roles of PDEs in image/signal analysis are only limited to PDE based low-pass filters, and their applications to noise removal, edge detection, segmentation, etc. At present, it is not clear how to construct PDE based methods for full-scale mode decomposition. The above-mentioned limitation of most current PDE based image/signal processing methods is addressed in the proposed work, in which we introduce a family of mode decomposition evolution equations (MoDEEs) for a vast variety of applications. The MoDEEs are constructed as an extension of a PDE based high-pass filter (Wei and Jia in Europhys. Lett. 59(6):814–819, 2002) by using arbitrarily high order PDE based low-pass filters introduced by Wei (IEEE Signal Process. Lett. 6(7):165–167, 1999). The use of arbitrarily high order PDEs is essential to the frequency localization in the mode decomposition. Similar to the wavelet transform, the present MoDEEs have a controllable time-frequency localization and allow a perfect reconstruction of the original function. Therefore, the MoDEE operation is also called a PDE transform. However, modes generated from the present approach are in the spatial or time domain and can be easily used for secondary processing. Various simplifications of the proposed MoDEEs, including a linearized version, and an algebraic version, are discussed for computational convenience. The Fourier pseudospectral method, which is unconditionally stable for linearized high order MoDEEs, is utilized in our computation. Validation is carried out to mode separation of high frequency adjacent modes. Applications are considered to signal and image denoising, image edge detection, feature extraction, enhancement etc. It is hoped that this work enhances the understanding of high order PDEs and yields robust and useful tools for image and signal analysis.     

Building Blocks for Computer Vision with Stochastic Partial Differential Equations

We discuss the basic concepts of computer vision with stochastic partial differential equations (SPDEs). In typical approaches based on partial differential equations (PDEs), the end result in the best case is usually one value per pixel, the “expected” value. Error estimates or even full probability density functions PDFs are usually not available. This paper provides a framework allowing one to derive such PDFs, rendering computer vision approaches into measurements fulfilling scientific standards due to full error propagation. We identify the image data with random fields in order to model images and image sequences which carry uncertainty in their gray values, e.g. due to noise in the acquisition process. The noisy behaviors of gray values is modeled as stochastic processes which are approximated with the method of generalized polynomial chaos (Wiener-Askey-Chaos). The Wiener-Askey polynomial chaos is combined with a standard spatial approximation based upon piecewise multi-linear finite elements. We present the basic building blocks needed for computer vision and image processing in this stochastic setting, i.e. we discuss the computation of stochastic moments, projections, gradient magnitudes, edge indicators, structure tensors, etc. Finally we show applications of our framework to derive stochastic analogs of well known PDEs for de-noising and optical flow extraction. These models are discretized with the stochastic Galerkin method. Our selection of SPDE models allows us to draw connections to the classical deterministic models as well as to stochastic image processing not based on PDEs. Several examples guide the reader through the presentation and show the usefulness of the framework.     

机器视觉应用中的图像数据增广综述

深度学习是目前机器视觉的前沿解决方案,而海量高质量的训练数据集是深度学习解决机器视觉问题的基本保障.收集和准确标注图像数据集是一个极其费时且代价昂贵的过程.随着机器视觉的广泛应用,这个问题将会越来越突出.图像增广技术是一种有效解决深度学习在少量或者低质量训练数据中进行训练的一种技术手段,该技术不断地伴随着深度学习与机器...     

Efficient solution of nonlinear,underdetermined inverse problems with a generalized PDE model

Several parameter estimation problems (or “inverse” problems) such as those that occur in hydrology and geophysics are solved using partial differential equation (PDE)-based models of the physical system in question. Likewise, these problems are usually underdetermined due to the lack of enough data to constrain a unique solution. In this paper, we present a framework for the solution of underdetermined inverse problems using COMSOL Multiphysics (formerly FEMLAB) that is applicable to a broad range of physical systems governed by PDEs. We present a general adjoint state formulation which may be used in this framework and allows for faster calculation of sensitivity matrices in a variety of commonly encountered underdetermined problems. The aim of this approach is to provide a platform for the solution of inverse problems that is efficient, flexible, and not restricted to one particular scientific application.We present an example application of this framework on a synthetic underdetermined inverse problem in aquifer characterization, and present numerical results on the accuracy and efficiency of this method. Our results indicate that our COMSOL-based routines provide an accurate, flexible, and scalable method for the solution of PDE-based inverse problems.     

张量值图像插值方法综述

在图像处理和计算机视觉的许多任务中,经常需要对图像进行插值从而得到像素点之间的信息。标量值图像的插值方法已经得到充分的发展,但张量值图像的插值方法还没有深刻的发展和认识。通过对比较零散的张量值图像插值方法的研究现状进行了系统综述,从数学理论框架的角度出发,将现有的张量值图像插值方法进行全面分析和分类,指出欧氏理论框架计算张量会带来的问题,梳理从欧氏框架到黎曼度量框架的研究脉络,并比较了张量值图像插值方法的评价指标。最后,给出了张量值图像插值方法未来研究方向的建议。     

计算机视觉下的果实目标检测算法综述

基于计算机视觉的果实目标检测识别是目标检测、计算机视觉、农业机器人等多学科的重要交叉研究课题,在智慧农业、农业现代化、自动采摘机器人等领域,具有重要的理论研究意义和实际应用价值。随着深度学习在图像处理领域中广泛应用并取得良好效果,计算机视觉技术结合深度学习方法的果实目标检测识别算法逐渐成为主流。本文介绍基于计算机视觉的果实目标检测识别的任务、难点和发展现状,以及2类基于深度学习方法的果实目标检测识别算法,最后介绍用于算法模型训练学习的公开数据集与评价模型性能的评价指标,且对当前果实目标检测识别存在的问题和未来可能的发展方向进行讨论。     

Surface processing methods for point sets using finite elements

We present a framework for processing point-based surfaces via partial differential equations (PDEs). Our framework efficiently and effectively brings well-known PDE-based processing techniques to the field of point-based surfaces. At the core of our method is a finite element discretization of PDEs on point surfaces. This discretization is based on the local assembly of PDE-specific mass and stiffness matrices, using a local point coupling computation. Point couplings are computed using a local tangent plane construction and a local Delaunay triangulation of point neighborhoods. The definition of tangent planes relies on moment-based computation with proven scaling and stability properties. Once local stiffness matrices are obtained, we are able to easily assemble global matrices and efficiently solve the corresponding linear systems by standard iterative solvers. We demonstrate our framework by several types of PDE-based surface processing applications, such as segmentation, texture synthesis, bump mapping, and geometric fairing.     

A VERSATILE DEVELOPMENT TOOL FOR KNOWLEDGE-BASED COMPUTER VISION

Knowledge-based computer vision creates a large variety of different design tasks including low-level image processing tasks, components for symbol manipulation, and complex cognitive processes. The diversity of requirements of these tasks calls for radically new concepts in terms of hardware and software structures. Classical design tools, such as image processing systems, support only small portions of this large task set. In this paper we report on an interactive and homogeneous software environment termed the Vision Kernel System (VKS) that aims at supporting the entire spectrum of problems encountered in knowledge-based computer vision.     

A shape design system using volumetric implicit PDEs

Solid modeling based on partial differential equations (PDEs) can potentially unify both geometric constraints and functional requirements within a single design framework to model real-world objects via its explicit, direct integration with parametric geometry. In contrast, implicit functions indirectly define geometric objects as the level-set of underlying scalar fields. To maximize the modeling potential of PDE-based methodology, in this paper we tightly couple PDEs with volumetric implicit functions in order to achieve interactive, intuitive shape representation, manipulation, and deformation. In particular, the unified approach can reconstruct the PDE geometry of arbitrary topology from scattered data points or a set of sketch curves. We make use of elliptic PDEs for boundary value problems to define the volumetric implicit function. The proposed implicit PDE model has the capability to reconstruct a complete solid model from partial information and facilitates the direct manipulation of underlying volumetric datasets via sketch curves and iso-surface sculpting, deformation of arbitrary interior regions, as well as a set of CSG operations inside the working space. The prototype system that we have developed allows designers to interactively sketch the curve outlines of the object, define intensity values and gradient directions, and specify interpolatory points in the 3D working space. The governing implicit PDE treats these constraints as generalized boundary conditions to determine the unknown scalar intensity values over the entire working space. The implicit shape is reconstructed with specified intensity value accordingly and can be deformed using a set of sculpting toolkits. We use the finite-difference discretization and variational interpolating approach with the localized iterative solver for the numerical integration of our PDEs in order to accommodate the diversity of generalized boundary and additional constraints.     

机器学习图像分类程序的蜕变测试框架

机器学习在计算机视觉、语音识别和自然语言处理等实际应用中已经取得了显著的成功。图像分类作为计算机视觉的一个主要分支。不久的将来,许多的图像分类程序会以机器学习的方式呈现。然而,由于机器学习图像分类程序的测试面临着测试预言难题,这使得在测试的过程中将需要大量的人力及物力。为了缓解测试预言难题,使用了蜕变测试技术。为了规范测试流程、提高测试效率,提出了一种适用于机器学习图像分类程序的蜕变测试框架。并且通过测试基于SVM和VGGNet图像分类程序,验证了该测试框架的合理性和有效性。     

Cyclic animation using partial differential equations

This work presents an efficient and fast method for achieving cyclic animation using partial differential equations (PDEs). The boundary-value nature associated with elliptic PDEs offers a fast analytic solution technique for setting up a framework for this type of animation. The surface of a given character is thus created from a set of pre-determined curves, which are used as boundary conditions so that a number of PDEs can be solved. Two different approaches to cyclic animation are presented here. The first of these approaches consists of attaching the set of curves to a skeletal system, which is responsible for holding the animation for cyclic motions through a set mathematical expressions. The second approach exploits the spine associated with the analytic solution of the PDE as a driving mechanism to achieve cyclic animation. The spine is also manipulated mathematically. In the interest of illustrating both approaches, the first one has been implemented within a framework related to cyclic motions inherent to human-like characters. Spine-based animation is illustrated by modelling the undulatory movement observed in fish when swimming. The proposed method is fast and accurate. Additionally, the animation can be either used in the PDE-based surface representation of the model or transferred to the original mesh model by means of a point to point map. Thus, the user is offered with the choice of using either of these two animation representations of the same object, the selection depends on the computing resources such as storage and memory capacity associated with each particular application.     

视觉语言预训练综述

近年来深度学习在计算机视觉(CV)和自然语言处理(NLP)等单模态领域都取得了十分优异的性能.随着技术的发展,多模态学习的重要性和必要性已经慢慢展现.视觉语言学习作为多模态学习的重要部分,得到国内外研究人员的广泛关注.得益于Transformer框架的发展,越来越多的预训练模型被运用到视觉语言多模态学习上,相关任务在性能上得到了质的飞跃.系统地梳理了当前视觉语言预训练模型相关的工作,首先介绍了预训练模型的相关知识,其次从两种不同的角度分析比较预训练模型结构,讨论了常用的视觉语言预训练技术,详细介绍了5类下游预训练任务,最后介绍了常用的图像和视频预训练任务的数据集,并比较和分析了常用预训练模型在不同任务下不同数据集上的性能.     

单幅图像去雨数据集和深度学习算法的联合评估与展望

雨天会影响室外图像捕捉的质量,进而引起户外视觉任务性能下降。基于深度学习的单幅图像去雨研究因算法性能优越而引起了大家的关注,并且聚焦点集中在数据集的质量、图像去雨方法、单幅图像去雨后续高层任务的研究和性能评价指标等方面。为了方便研究者快速全面了解该领域,本文从上述4个方面综述了基于深度学习的单幅图像去雨的主流文献。依据数据集的构建方式将雨图数据集分为4类：基于背景雨层简单加和、背景雨层复杂融合、生成对抗网络 (generative adversarial network,GAN)数据驱动合成的数据集,以及半自动化采集的真实数据集。依据任务场景、采取的学习机制以及网络设计对主流算法分类总结。综述了面向单任务和联合任务的去雨算法,单任务即雨滴、雨纹、雨雾和暴雨的去除;联合任务即雨滴和雨纹、所有噪声去除。综述了学习机制和网络构建方式(比如:卷积神经网络 (convolutional neural network,CNN)结构多分支组合,GAN的生成结构,循环和多阶段结构,多尺度结构,编解码结构,基于注意力,基于Transformer)以及数据模型双驱动的构建方式。综述了单幅图像去雨后续高层任务的研究文献和图像去雨算法性能的评价指标。通过合成数据集和真实数据集上的综合实验对比,证实了领域知识隐式引导网络构建可以有效提升算法性能,领域知识显式引导正则化网络的学习有潜力进一步提升算法的泛化性。最后,指出单幅图像去雨工作目前面临的挑战和未来的研究方向。     

Free-form geometric modeling by integrating parametric and implicit PDEs

Parametric PDE techniques, which use partial differential equations (PDEs) defined over a 2D or 3D parametric domain to model graphical objects and processes, can unify geometric attributes and functional constraints of the models. PDEs can also model implicit shapes defined by level sets of scalar intensity fields. In this paper, we present an approach that integrates parametric and implicit trivariate PDEs to define geometric solid models containing both geometric information and intensity distribution subject to flexible boundary conditions. The integrated formulation of second-order or fourth-order elliptic PDEs permits designers to manipulate PDE objects of complex geometry and/or arbitrary topology through direct sculpting and free-form modeling. We developed a PDE-based geometric modeling system for shape design and manipulation of PDE objects. The integration of implicit PDEs with parametric geometry offers more general and arbitrary shape blending and free-form modeling for objects with intensity attributes than pure geometric models     

Dynamic PDE-based surface design using geometric and physical constraints

PDE surfaces, which are defined as solutions of partial differential equations (PDEs), offer many modeling advantages in surface blending, free-form surface modeling, and specifying surface’s aesthetic or functional requirements. Despite the earlier advances of PDE surfaces, previous PDE-based techniques exhibit certain difficulties such as lack of interactive sculpting capabilities and restrained topological structure of modeled objects. This paper presents an integrated approach that can incorporate PDE surfaces into the powerful physics-based modeling framework, to realize the full potential of PDE methodology. We have developed a prototype system that allows interactive design of flexible topological surfaces as PDE surfaces and displacements using generalized boundary conditions as well as a variety of geometric and physical constraints, hence supporting various interactive techniques beyond the conventional boundary control. The system offers a set of sculpting toolkits that allow users to interactively modify arbitrary points, curve spans, and/or regions of interest across the entire PDE surfaces and displacements in an intuitive and physically meaningful way. To achieve real-time performance, we employ several simple, yet efficient numerical techniques, including the finite-difference discretization, the multigrid-like subdivision, and the mass-spring approximation of elastic PDE surfaces and displacements. In addition, we present the standard bivariant B-spline finite element approximations of dynamic PDEs, which can subsequently be sculpted and deformed directly in real-time subject to the intrinsic PDE constraints. Our experiments demonstrate many attractive advantages of the physics-based PDE formulation such as intuitive control, real-time feedback, and usability to both professional and common users.     

Deep topology network: A framework based on feedback adjustment learning rate for image classification

Convolutional Neural Network (CNN) has demonstrated its superior ability to achieve amazing accuracy in computer vision field. However, due to the limitation of network depth and computational complexity, it is still difficult to obtain the best classification results for the specific image classification tasks. In order to improve classification performance without increasing network depth, a new Deep Topology Network (DTN) framework is proposed. The key idea of DTN is based on the iteration of multiple learning rate feedback. The framework consists of multiple sub-networks and each sub-network has its own learning rate. After the determined iteration period, these learning rates can be adjusted according to the feedback of training accuracy, in the feature learning process, the optimal learning rate is updated iteratively to optimize the loss function. In practice, the proposed DTN framework is applied to several state-of-the-art deep networks, and its performance is tested by extensive experiments and comprehensive evaluations of CIFAR-10 and MNIST benchmarks. Experimental results show that most deep networks can benefit from the DTN framework with an accuracy of 99.5% on MINIST dataset, which is 5.9% higher than that on the CIFAR-10 benchmark.     

A survey of Optimal Transport for Computer Graphics and Computer Vision

Optimal transport is a long-standing theory that has been studied in depth from both theoretical and numerical point of views. Starting from the 50s this theory has also found a lot of applications in operational research. Over the last 30 years it has spread to computer vision and computer graphics and is now becoming hard to ignore. Still, its mathematical complexity can make it difficult to comprehend, and as such, computer vision and computer graphics researchers may find it hard to follow recent developments in their field related to optimal transport. This survey first briefly introduces the theory of optimal transport in layman's terms as well as most common numerical techniques to solve it. More importantly, it presents applications of these numerical techniques to solve various computer graphics and vision related problems. This involves applications ranging from image processing, geometry processing, rendering, fluid simulation, to computational optics, and many more. It is aimed at computer graphics researchers desiring to follow optimal transport research in their field as well as optimal transport researchers willing to find applications for their numerical algorithms.     

A Framework for Robust Subspace Learning

Many computer vision, signal processing and statistical problems can be posed as problems of learning low dimensional linear or multi-linear models. These models have been widely used for the representation of shape, appearance, motion, etc., in computer vision applications. Methods for learning linear models can be seen as a special case of subspace fitting. One draw-back of previous learning methods is that they are based on least squares estimation techniques and hence fail to account for outliers which are common in realistic training sets. We review previous approaches for making linear learning methods robust to outliers and present a new method that uses an intra-sample outlier process to account for pixel outliers. We develop the theory of Robust Subspace Learning (RSL) for linear models within a continuous optimization framework based on robust M-estimation. The framework applies to a variety of linear learning problems in computer vision including eigen-analysis and structure from motion. Several synthetic and natural examples are used to develop and illustrate the theory and applications of robust subspace learning in computer vision.     

Numerical optimal control of the wave equation: optimal boundary control of a string to rest in finite time

In many real-life applications of optimal control problems with constraints in form of partial differential equations (PDEs), hyperbolic equations are involved which typically describe transport processes. Since hyperbolic equations usually propagate discontinuities of initial or boundary conditions into the domain on which the solution lives or can develop discontinuities even in the presence of smooth data, problems of this type constitute a severe challenge for both theory and numerics of PDE constrained optimization.     

Formulating efficient software solution for digital image processing system

Digital image processing systems are complex, being usually composed of different computer vision libraries. Algorithm implementations cannot be directly used in conjunction with algorithms developed using other computer vision libraries. This paper formulates a software solution by proposing a processor with the capability of handling different types of image processing algorithms, which allow the end users to install new image processing algorithms from any library. This approach has other functionalities like capability to process one or more images, manage multiple processing jobs simultaneously and maintain the manner in which an image was processed for later use. It is a computational efficient and promising technique to handle variety of image processing algorithms. To promote the reusability and adaptation of the package for new types of analysis, a feature of sustainability is established. The framework is integrated and tested on a medical imaging application, and the software is made freely available for the reader. Future work involves introducing the capability to connect to another instance of processing service with better performance. Copyright © 2015 John Wiley & Sons, Ltd.