Deep cross-view autoencoder network for multi-view learning

In many real-world applications, an increasing number of objects can be collected at varying viewpoints or by different sensors, which brings in the urgent demand for recognizing objects from distinct heterogeneous views. Although significant progress has been achieved recently, heterogeneous recognition (cross-view recognition) in multi-view learning is still challenging due to the complex correlations among views. Multi-view subspace learning is an effective solution, which attempts to obtain a common representation from downstream computations. Most previous methods are based on the idea of maximal correlation after feature extraction to establish the relationship among different views in a two-step manner, thus leading to performance deterioration. To overcome this drawback, in this paper, we propose a deep cross-view autoencoder network (DCVAE) that extracts the features of different views and establishes the correlation between views in one step to simultaneously handle view-specific, view-correlation, and consistency in a joint manner. Specifically, DCVAE contains self-reconstruction, newly designed cross-view reconstruction, and consistency constraint modules. Self-reconstruction ensures the view-specific, cross-view reconstruction transfers the information from one view to another view, and consistency constraint makes the representation of different views more consistent. The proposed model suffices to discover the complex correlation embedded in multi-view data and to integrate heterogeneous views into a latent common representation subspace. Furthermore, the 2D embeddings of the learned common representation subspace demonstrate the consistency constraint is valid and cross-view classification experiments verify the superior performance of DCVAE in the two-view scenario.

     

IBMvSVM: An instance-based multi-view SVM algorithm for classification

As an emerging research direction of machine learning, the multi-view learning (MVL) pays attention to the tasks that learn from datasets with several distinct views to achieve better generalization performance. Recently, various Support Vector Machine (SVM)-based algorithms with solid theoretical foundation have been proposed for MVL. However, there is a constraining assumption for these algorithms, i.e., in the learning process, different views are important equally for an instance in a data set, and the same view is important equally for all instances in a data set. In fact, an instance generally has different adaptability to different views, namely, the degree to which the information from different views accurately describes the instance varies. And naturally, different instances in a data set also have different adaptability to the same view. In this paper, the concept of view vector of each instance is proposed first, which quantitatively describes the adaptability of a specific instance to different views. It also reflects the characteristics of different instances that some instances are more suitable to be represented by a view, while others tend to be better represented by another view. Then, a new instance-based multi-view SVM algorithm, named IBMvSVM, is proposed by building the view vector of each instance into the multi-view SVM learning. IBMvSVM focuses on characteristics of each instance itself in different views rather than treating them equally. Experiments performed on 48 multi-view datasets reveal the superiority of IBMvSVM algorithm on generalization against several recently state-of-the-art MVL algorithms.

     

A novel multi-view learning developed from single-view patterns

The existing multi-view learning (MVL) learns how to process patterns with multiple information sources. In generalization this MVL is proven to have a significant advantage over the usual single-view learning (SVL). However, in most real-world cases we only have single source patterns to which the existing MVL is unable to be directly applied. This paper aims to develop a new MVL technique for single source patterns. To this end, we first reshape the original vector representation of single source patterns into multiple matrix representations. In doing so, we can change the original architecture of a given base classifier into different sub-ones. Each newly generated sub-classifier can classify the patterns represented with the matrix. Here each sub-classifier is taken as one view of the original base classifier. As a result, a set of sub-classifiers with different views are come into being. Then, one joint rather than separated learning process for the multi-view sub-classifiers is developed. In practice, the original base classifier employs the vector-pattern-oriented Ho–Kashyap classifier with regularization learning (called MHKS) as a paradigm which is not limited to MHKS. Thus, the proposed joint multi-view learning is named as MultiV-MHKS. Finally, the feasibility and effectiveness of the proposed MultiV-MHKS is demonstrated by the experimental results on benchmark data sets. More importantly, we have demonstrated that the proposed multi-view approach generally has a tighter generalization risk bound than its single-view one in terms of the Rademacher complexity analysis.     

基于多视角图编码的选择式阅读理解方法

选择式阅读理解通常采用证据抽取和答案预测的两阶段流水线框架,答案预测的效果非常依赖于证据句抽取的效果.传统的证据抽取多依赖词段匹配或利用噪声标签监督证据抽取的方法,准确率不理想,这极大地影响了答案预测的性能.针对该问题,提出一种联合学习框架下基于多视角图编码的选择式阅读理解方法,从多视角充分挖掘文档句子之间以及文档句子和问句之间的关联关系,实现证据句及其关系的有效建模;同时通过联合训练证据抽取和答案预测任务,利用证据和答案之间强关联关系提升证据抽取与答案预测的性能.具体来说,所提方法首先基于多视角图编码模块对文档、问题和候选答案联合编码,从统计特性、相对距离和深度语义3个视角捕捉文档、问题和候选答案之间的关系,获得问答对感知的文档编码特征;然后,构建证据抽取和答案预测的联合学习模块,通过协同训练强化证据与答案之间的关系,证据抽取子模块实现证据句的选择,并将其结果和文档编码特征进行选择性融合,并用于答案预测子模块完成答案预测.在选择式阅读理解数据集ReCO和RACE上的实验结果表明,所提方法提升了从文档中选择证据句子的能力,进而提高答案预测的准确率.同时,证据抽取与答案预测联合学习很大程...     

Robust multi-view feature matching from multiple unordered views

This paper explores the problem of multi-view feature matching from an unordered set of widely separated views. A set of local invariant features is extracted independently from each view. First we propose a new view-ordering algorithm that organizes all the unordered views into clusters of related (i.e. the same scene) views by efficiently computing the view-similarity values of all view pairs by reasonably selecting part of extracted features to match. Second a robust two-view matching algorithm is developed to find initial matches, then detect the outliers and finally incrementally find more reliable feature matches under the epipolar constraint between two views from dense to sparse based on an assumption that changes of both motion and feature characteristics of one match are consistent with those of neighbors. Third we establish the reliable multi-view matches across related views by reconstructing missing matches in a neighboring triple of views and efficiently determining the states of matches between view pairs. Finally, the reliable multi-view matches thus obtained are used to automatically track all the views by using a self-calibration method. The proposed methods were tested on several sets of real images. Experimental results show that it is efficient and can track a large set of multi-view feature matches across multiple widely separated views.     

Tensor-based restricted kernel machines for multi-view classification

Multi-view learning deals with data that is described through multiple representations, or views. While various real-world data can be represented by three or more views, several existing multi-view classification methods can only handle two views. Previously proposed methods usually solve this issue by optimizing pairwise combinations of views. Although this can numerically deal with the issue of multiple views, it ignores the higher order correlations which can only be examined by exploring all views simultaneously. In this work new multi-view classification approaches are introduced which aim to include higher order statistics when three or more views are available. The proposed model is an extension to the recently proposed Restricted Kernel Machine classifier model and assumes shared hidden features for all views, as well as a newly introduced model tensor. Experimental results show an improvement with respect to state-of-the art pairwise multi-view learning methods, both in terms of classification accuracy and runtime.     

A novel ensemble construction method for multi-view data using random cross-view correlation between within-class examples

Correlated information between multiple views can provide useful information for building robust classifiers. One way to extract correlated features from different views is using canonical correlation analysis (CCA). However, CCA is an unsupervised method and can not preserve discriminant information in feature extraction. In this paper, we first incorporate discriminant information into CCA by using random cross-view correlations between within-class examples. Because of the random property, we can construct a lot of feature extractors based on CCA and random correlation. So furthermore, we fuse those feature extractors and propose a novel method called random correlation ensemble (RCE) for multi-view ensemble learning. We compare RCE with existing multi-view feature extraction methods including CCA and discriminant CCA (DCCA) which use all cross-view correlations between within-class examples, as well as the trivial ensembles of CCA and DCCA which adopt standard bagging and boosting strategies for ensemble learning. Experimental results on several multi-view data sets validate the effectiveness of the proposed method.     

mPadal: a joint local-and-global multi-view feature selection method for activity recognition

The selection of multi-view features plays an important role for classifying multi-view data, especially the data with high dimension. In this paper, a novel multi-view feature selection method via joint local pattern-discrimination and global label-relevance analysis (mPadal) is proposed. Different from the previous methods which globally select the multi-view features directly via view-level analysis, the proposed mPadal employs a new joint local-and-global way. In the local selection phase, the pattern-discriminative features will be first selected by considering the local neighbor structure of the most discriminative patterns. In the global selection phase, the features with the topmost label-relevance, which can well separate different classes in the current view, are selected. Finally, the two parts selected are combined to form the final features. Experimental results show that compared with several baseline methods in publicly available activity recognition dataset IXMAS, mPadal performs the best in terms of the highest accuracy, precision, recall and F1 score. Moreover, the features selected by mPadal are highly complementary among views for classification, which is able to improve the classification performance according to previous theoretical studies.     

基于块对角化表示的多视角字典对学习

字典学习作为一种高效的特征学习技术被广泛应用于多视角分类中.现有的多视角字典学习方法大多只利用多视角数据的部分信息,且只学习一种类型的字典.实际上,多视角数据的相关性信息和多样性信息同样重要,且仅考虑一种合成型字典或解析型字典的学习算法不能同时满足处理速度、可解释性以及应用范围的要求.针对上述问题,提出了一种基于块对角...     

Multi-view document clustering via ensemble method

Multi-view clustering has become an important extension of ensemble clustering. In multi-view clustering, we apply clustering algorithms on different views of the data to obtain different cluster labels for the same set of objects. These results are then combined in such a manner that the final clustering gives better result than individual clustering of each multi-view data. Multi view clustering can be applied at various stages of the clustering paradigm. This paper proposes a novel multi-view clustering algorithm that combines different ensemble techniques. Our approach is based on computing different similarity matrices on the individual datasets and aggregates these to form a combined similarity matrix, which is then used to obtain the final clustering. We tested our approach on several datasets and perform a comparison with other state-of-the-art algorithms. Our results show that the proposed algorithm outperforms several other methods in terms of accuracy while maintaining the overall complexity of the individual approaches.     

Multi-view constrained clustering with an incomplete mapping between views

Multi-view learning algorithms typically assume a complete bipartite mapping between the different views in order to exchange information during the learning process. However, many applications provide only a partial mapping between the views, creating a challenge for current methods. To address this problem, we propose a multi-view algorithm based on constrained clustering that can operate with an incomplete mapping. Given a set of pairwise constraints in each view, our approach propagates these constraints using a local similarity measure to those instances that can be mapped to the other views, allowing the propagated constraints to be transferred across views via the partial mapping. It uses co-EM to iteratively estimate the propagation within each view based on the current clustering model, transfer the constraints across views, and then update the clustering model. By alternating the learning process between views, this approach produces a unified clustering model that is consistent with all views. We show that this approach significantly improves clustering performance over several other methods for transferring constraints and allows multi-view clustering to be reliably applied when given a limited mapping between the views. Our evaluation reveals that the propagated constraints have high precision with respect to the true clusters in the data, explaining their benefit to clustering performance in both single- and multi-view learning scenarios.     

Adaptive partial graph learning and fusion for incomplete multi-view clustering

Most of existing multi-view clustering methods assume that different feature views of data are fully observed. However, it is common that only portions of data features can be obtained in many practical applications. The presence of incomplete feature views hinders the performance of the conventional multi-view clustering methods to a large extent. Recently proposed incomplete multi-view clustering methods often focus on directly learning a common representation or a consensus affinity similarity graph from available feature views while ignore the valuable information hidden in the missing views. In this study, we present a novel incomplete multi-view clustering method via adaptive partial graph learning and fusion (APGLF), which can capture the local data structure of both within-view and cross-view. Specifically, we use the available data of each view to learn a corresponding view-specific partial graph, in which the within-view local structure can be well preserved. Then we design a cross-view graph fusion term to learn a consensus complete graph for different views, which can take advantage of the complementary information hidden in the view-specific partial graphs learned from incomplete views. In addition, a rank constraint is imposed on the graph Laplacian matrix of the fused graph to better recover the optimal cluster structure of original data. Therefore, APGLF integrates within-view partial graph learning, cross-view partial graph fusion and cluster structure recovering into a unified framework. Experiments on five incomplete multi-view data sets are conducted to validate the efficacy of APGLF when compared with eight state-of-the-art methods.     

Multi-view k-proximal plane clustering

Multi-view clustering is an active direction in machine learning and pattern recognition which aims at exploring the consensus and complementary information among multiple views. In the last few years, a number of methods based on multi-view learning have been widely investigated and achieved promising performance. Generally, classical multi-view clustering methods such as multi-view kernel k-means clustering are point-based methods. The performance of point-based methods will be fairly good when the data points are distributed around the center point. The plane-based clustering methods can handle data points that are clustered along a straight line and have never been investigated in multi-view learning. In this paper, we propose a novel multi-view k-proximal plane clustering method, which initializes cluster labels by multi-view spectralclustering and updates whole multi-view cluster hyperplanes and labels alternately until some stopping conditions are satisfied. Extensive experimental results on several benchmark datasets show that the proposed model outperforms other state-of-the-art multi-view algorithms.

     

Sparse points matching by combining 3D mesh saliency with statistical descriptors

This paper proposes new methodology for the detection and matching of salient points over several views of an object. The process is composed by three main phases. In the first step, detection is carried out by adopting a new perceptually‐inspired 3D saliency measure. Such measure allows the detection of few sparse salient points that characterize distinctive portions of the surface. In the second step, a statistical learning approach is considered to describe salient points across different views. Each salient point is modelled by a Hidden Markov Model (HMM), which is trained in an unsupervised way by using contextual 3D neighborhood information, thus providing a robust and invariant point signature. Finally, in the third step, matching among points of different views is performed by evaluating a pairwise similarity measure among HMMs. An extensive and comparative experimental session has been carried out, considering real objects acquired by a 3D scanner from different points of view, where objects come from standard 3D databases. Results are promising, as the detection of salient points is reliable, and the matching is robust and accurate.     

基于稀疏学习的行人重识别算法

行人重识别问题是计算机视觉的重要研究内容之一,旨在将多个非重叠相机中的目标行人准确加以识别。当将某摄像机中的行人图像视为目标行人在该摄像机视图上的一种表示时,行人重识别可被认为是一种多视图学习问题。在此基础上提出的基于典型相关分析的行人重识别算法仅是一种线性降维算法,很难从复杂的重识别系统（如目标行人图像受低分辨率、光照及行人姿态变化等因素影响）中提取有效的高层语义信息,用于行人重识别。为此,本文提出了一种基于稀疏学习的行人重识别算法（Sparsity learning based person re-identification,SLR）。SLR首先通过稀疏学习获取目标行人在每一相机视图上的高层语义表示,然后将高层特征映射到一个公共的隐空间,使不同视图间的特征距离可比较。SLR算法的优点在于通过学习鲁棒的行人图像特征表示,能够获得更具判别性的公共隐空间,以提高算法的行人重识别性能。在VIPeR、CUHK数据集上的实验结果表明了本文算法的有效性。     

Strategies of Multi-view and Multi-matching for 3D Object Recognition

A multi-view representation scheme and a multi-matching strategy for 3D object recognition are described; 3D objects are represented in terms of their 2D appearances so that 2D techniques can be applied to 3D recognition. Appearances of objects in the representation scheme are further organized in a hierarchical manner so that the matching process can reduce its search space by examining only the optimal view at every level of the representation scheme. In our multi-matching strategy, the matching module is composed of four components: point matcher, string matcher, vector matcher, and chamfer matcher. Each matcher is associated with a termination rule so that impossible views can be rejected at the early stages of the matching process. Experimental results reveal that the proposed strategies are feasible for 3D object recognition.     

Kernelized multi-view subspace clustering via auto-weighted graph learning

Multi-view subspace clustering has been an important and powerful tool for partitioning multi-view data, especially multi-view high-dimensional data. Despite great success, most of the existing multi-view subspace clustering methods still suffer from three limitations. First, they often recover the subspace structure in the original space, which can not guarantee the robustness when handling multi-view data with nonlinear structure. Second, these methods mostly regard subspace clustering and affinity matrix learning as two independent steps, which may not well discover the latent relationships among data samples. Third, many of them ignore the different importance of multiple views, whose performance may be badly affected by the low-quality views in multi-view data. To overcome these three limitations, this paper develops a novel subspace clustering method for multi-view data, termed Kernelized Multi-view Subspace Clustering via Auto-weighted Graph Learning (KMSC-AGL). Specifically, the proposed method implicitly maps the multi-view data from linear space into nonlinear space via kernel-induced functions, so as to exploit the nonlinear structure hidden in data. Furthermore, our method aims to enhance the clustering performance by learning a set of view-specific representations and their affinity matrix in a general framework. By integrating the view weighting strategy into this framework, our method can automatically assign the weights to different views, while learning an optimal affinity matrix that is well-adapted to the subsequent spectral clustering. Extensive experiments are conducted on a variety of multi-view data sets, which have demonstrated the superiority of the proposed method.

     

鲁棒多视图协同完整鉴别子空间学习算法

为了有效地融合多视图信息并使有利于多视图完整子空间学习的视图主导多视图学习,提出了多视图协同完整子空间学习策略。进一步,为了使对象在潜在完整子空间中的完整特征表示具有更好的鉴别能力,将Fisher鉴别分析引入到了多视图完整子空间学习中。Fisher鉴别分析可以在最小化对象的完整特征表示的类内散度的同时最大化对象的完整特征表示的类间散度。将多视图协同完整空间学习策略和Fisher鉴别分析融合在一起,提出了鲁棒多视图协同完整鉴别子空间学习算法。实验结果表明,所提算法能够有效地融合多视图信息并挖掘鉴别信息,是一种有效的多视图完整子空间学习算法。     

Multi-view stereo in the Deep Learning Era: A comprehensive review

Multi-view stereo infers the 3D geometry from a set of images captured from several known positions and viewpoints. It is one of the most important components of 3D reconstruction. Recently, deep learning has been increasingly used to solve several 3D vision problems due to the predominating performance, including the multi-view stereo problem. This paper presents a comprehensive review, covering recent deep learning methods for multi-view stereo. These methods are mainly categorized into depth map based and volumetric based methods according to the 3D representation form, and representative methods are reviewed in detail. Specifically, the plane sweep based methods leveraging depth maps are presented following the stage of approaches, i.e. feature extraction, cost volume construction, cost volume regularization, depth map regression and post-processing. This review also summarizes several widely used datasets and their corresponding metrics for evaluation. Finally, several insightful observations and challenges are put forward enlightening future research directions.     

基于生成对抗网络的多视图学习与重构算法

同一事物通常需要从不同角度进行表达.然而,现实应用经常引出复杂的场景,导致完整视图数据很难获得.因此研究如何构建事物的完整视图具有重要意义.本文提出一种基于生成对抗网络（Generative adversarial networks,GAN）的多视图学习与重构算法,利用已知单一视图,通过生成式方法构建其他视图.为构建多视图通用的表征,提出新型表征学习算法,使得同一实例的任意视图都能映射至相同的表征向量,并保证其包含实例的重构信息.为构建给定事物的多种视图,提出基于生成对抗网络的重构算法,在生成模型中加入表征信息,保证了生成视图数据与源视图相匹配.所提出的算法的优势在于避免了不同视图间的直接映射,解决了训练数据视图不完整问题,以及构造视图与已知视图正确对应问题.在手写体数字数据集MNIST,街景数字数据集SVHN和人脸数据集CelebA上的模拟实验结果表明,所提出的算法具有很好的重构性能.