Matching samples of multiple views

Multi-view learning studies how several views, different feature representations, of the same objects could be best utilized in learning. In other words, multi-view learning is analysis of co-occurrence data, where the observations are co-occurrences of samples in the views. Standard multi-view learning such as joint density modeling cannot be done in the absence of co-occurrence, when the views are observed separately and the identities of objects are not known. As a practical example, joint analysis of mRNA and protein concentrations requires mapping between genes and proteins. We introduce a data-driven approach for learning the correspondence of the observations in the different views, in order to enable joint analysis also in the absence of known co-occurrence. The method finds a matching that maximizes statistical dependency between the views, which is particularly suitable for multi-view methods such as canonical correlation analysis which has the same objective. We apply the method to translational metabolomics, to identify differences and commonalities in metabolic processes in different species or tissues. The metabolite identities and roles in the different species are not generally known, and it is necessary to search for a matching. In this paper we show, using different metabolomics measurement batches as the views so that the ground truth is known, that the metabolite identities can be reliably matched by a consensus of several matching solutions.     

基于深度卷积神经网络的三维模型检索

为了更有效地利用三维模型数据集进行特征的自主学习,提出一种使用自然图像作为输入源,以三维模型的较优视图集为基础,通过深度卷积神经网络的训练提取深度特征用于检索的三维模型检索方法。首先,从多个视点对三维模型进行视图提取,并根据灰度熵的排序选取较优视图;然后,通过深度卷积神经网络对视图集进行训练,从而提取较优视图的深度特征并进行降维,同时,对输入的自然图像提取边缘轮廓图,经过相似度匹配获得一组三维模型;最后,基于检索结果中同类模型总数占检索列表长度的比例对列表进行重排序,从而获得最终的检索结果。实验结果表明,该算法能够有效利用深度卷积神经网络对三维模型的视图进行深度特征提取,同时降低了输入源的获取难度,有效提高了检索效果。     

基于多模态融合的三维模型检索算法研究

为了获得更好的三维模型检索分类性能,基于深度学习模型研究了多模态信息融合对三维模型的特征描述,在训练步骤提出相关性损失函数来指导不同模态之间的训练,提取更稳健的特征向量;最后将融合特征应用于三维模型的检索和分类,在ModelNet40数据集上进行了三维模型分类任务和检索任务评估。实验结果及与现有方法进行的对比证明了该方法的优越性,为三维模型检索分类领域提供了一种新的思路。     

Efficient 2D viewpoint combination for human action recognition

The ability to recognize human actions using a single viewpoint is affected by phenomena such as self-occlusions or occlusions by other objects. Incorporating multiple cameras can help overcome these issues. However, the question remains how to efficiently use information from all viewpoints to increase performance. Researchers have reconstructed a 3D model from multiple views to reduce dependency on viewpoint, but this 3D approach is often computationally expensive. Moreover, the quality of each view influences the overall model and the reconstruction is limited to volumes where the views overlap. In this paper, we propose a novel method to efficiently combine 2D data from different viewpoints. Spatio-temporal features are extracted from each viewpoint and then used in a bag-of-words framework to form histograms. Two different sizes of codebook are exploited. The similarity between the obtained histograms is represented via the Histogram Intersection kernel as well as the RBF kernel with \(\chi ^2\) distance. Lastly, we combine all the basic kernels generated by selection of different viewpoints, feature types, codebook sizes and kernel types. The final kernel is a linear combination of basic kernels that are properly weighted based on an optimization process. For higher accuracy, the sets of kernel weights are computed separately for each binary SVM classifier. Our method not only combines the information from multiple viewpoints efficiently, but also improves the performance by mapping features into various kernel spaces. The efficiency of the proposed method is demonstrated by testing on two commonly used multi-view human action datasets. Moreover several experiments indicate the efficacy of each part of the method on the overall performance.     

自适应嵌入的半监督多视角特征降维方法

半监督模式下的多视角特征降维方法,大多并未考虑到不同视角间特征投影的差异,且由于缺乏对降维后的低维矩阵的稀疏约束,无法避免噪声和其他不相关特征的影响。针对这两个问题,提出自适应嵌入的半监督多视角特征降维方法。首先,将投影从单视角下相同的嵌入矩阵扩展到多视角间不同的矩阵,引入全局结构保持项;然后,将无标签的数据利用无监督方法进行嵌入投影,对于有标签的数据,结合分类的判别信息进行线性投影;最后,再将两类多投影映射到统一的低维空间,使用组合权重矩阵来保留全局结构,很大程度上消除了噪声及不相关因素的影响。实验结果表明,所提方法的聚类准确率平均提高了约9%。该方法较好地保留了多视角间特征的相关性,捕获了更多的具有判别信息的特征。     

Generative multi-view and multi-feature learning for classification

Multi-view based classification has attracted much attention in recent years. In general, an object can be represented with various views or modalities, and the exploitation of correlation across different views would contribute to improving the classification performance. However, each view can also be described with multiple features and this types of data is called multi-view and multi-feature data. Different from many existing multi-view methods which only model multiple views but ignore intrinsic information among the various features in each view, a generative bayesian model is proposed in this paper to not only jointly take the features and views into account, but also learn a discriminant representation across distinctive categories. A latent variable corresponding to each feature in each view is assumed and the raw feature is a projection of the latent variable from a more discriminant space. Particularly, the extracted variables in each view belonging to the same class are encouraged to follow the same gaussian distribution and those belonging to different classes are conducted to follow different distributions, greatly exploiting the label information. To optimize the presented approach, the proposed method is transformed into a class-conditional model and an effective algorithm is designed to alternatively estimate the parameters and variables. The experimental results on the extensive synthetic and four real-world datasets illustrate the effectiveness and superiority of our method compared with the state-of-the-art.     

A novel ensemble construction method for multi-view data using random cross-view correlation between within-class examples

Correlated information between multiple views can provide useful information for building robust classifiers. One way to extract correlated features from different views is using canonical correlation analysis (CCA). However, CCA is an unsupervised method and can not preserve discriminant information in feature extraction. In this paper, we first incorporate discriminant information into CCA by using random cross-view correlations between within-class examples. Because of the random property, we can construct a lot of feature extractors based on CCA and random correlation. So furthermore, we fuse those feature extractors and propose a novel method called random correlation ensemble (RCE) for multi-view ensemble learning. We compare RCE with existing multi-view feature extraction methods including CCA and discriminant CCA (DCCA) which use all cross-view correlations between within-class examples, as well as the trivial ensembles of CCA and DCCA which adopt standard bagging and boosting strategies for ensemble learning. Experimental results on several multi-view data sets validate the effectiveness of the proposed method.     

Deep cross-view autoencoder network for multi-view learning

In many real-world applications, an increasing number of objects can be collected at varying viewpoints or by different sensors, which brings in the urgent demand for recognizing objects from distinct heterogeneous views. Although significant progress has been achieved recently, heterogeneous recognition (cross-view recognition) in multi-view learning is still challenging due to the complex correlations among views. Multi-view subspace learning is an effective solution, which attempts to obtain a common representation from downstream computations. Most previous methods are based on the idea of maximal correlation after feature extraction to establish the relationship among different views in a two-step manner, thus leading to performance deterioration. To overcome this drawback, in this paper, we propose a deep cross-view autoencoder network (DCVAE) that extracts the features of different views and establishes the correlation between views in one step to simultaneously handle view-specific, view-correlation, and consistency in a joint manner. Specifically, DCVAE contains self-reconstruction, newly designed cross-view reconstruction, and consistency constraint modules. Self-reconstruction ensures the view-specific, cross-view reconstruction transfers the information from one view to another view, and consistency constraint makes the representation of different views more consistent. The proposed model suffices to discover the complex correlation embedded in multi-view data and to integrate heterogeneous views into a latent common representation subspace. Furthermore, the 2D embeddings of the learned common representation subspace demonstrate the consistency constraint is valid and cross-view classification experiments verify the superior performance of DCVAE in the two-view scenario.

     

基于块对角化表示的多视角字典对学习

字典学习作为一种高效的特征学习技术被广泛应用于多视角分类中.现有的多视角字典学习方法大多只利用多视角数据的部分信息,且只学习一种类型的字典.实际上,多视角数据的相关性信息和多样性信息同样重要,且仅考虑一种合成型字典或解析型字典的学习算法不能同时满足处理速度、可解释性以及应用范围的要求.针对上述问题,提出了一种基于块对角...     

Robust multi-view feature matching from multiple unordered views

This paper explores the problem of multi-view feature matching from an unordered set of widely separated views. A set of local invariant features is extracted independently from each view. First we propose a new view-ordering algorithm that organizes all the unordered views into clusters of related (i.e. the same scene) views by efficiently computing the view-similarity values of all view pairs by reasonably selecting part of extracted features to match. Second a robust two-view matching algorithm is developed to find initial matches, then detect the outliers and finally incrementally find more reliable feature matches under the epipolar constraint between two views from dense to sparse based on an assumption that changes of both motion and feature characteristics of one match are consistent with those of neighbors. Third we establish the reliable multi-view matches across related views by reconstructing missing matches in a neighboring triple of views and efficiently determining the states of matches between view pairs. Finally, the reliable multi-view matches thus obtained are used to automatically track all the views by using a self-calibration method. The proposed methods were tested on several sets of real images. Experimental results show that it is efficient and can track a large set of multi-view feature matches across multiple widely separated views.     

Multi-view constrained clustering with an incomplete mapping between views

Multi-view learning algorithms typically assume a complete bipartite mapping between the different views in order to exchange information during the learning process. However, many applications provide only a partial mapping between the views, creating a challenge for current methods. To address this problem, we propose a multi-view algorithm based on constrained clustering that can operate with an incomplete mapping. Given a set of pairwise constraints in each view, our approach propagates these constraints using a local similarity measure to those instances that can be mapped to the other views, allowing the propagated constraints to be transferred across views via the partial mapping. It uses co-EM to iteratively estimate the propagation within each view based on the current clustering model, transfer the constraints across views, and then update the clustering model. By alternating the learning process between views, this approach produces a unified clustering model that is consistent with all views. We show that this approach significantly improves clustering performance over several other methods for transferring constraints and allows multi-view clustering to be reliably applied when given a limited mapping between the views. Our evaluation reveals that the propagated constraints have high precision with respect to the true clusters in the data, explaining their benefit to clustering performance in both single- and multi-view learning scenarios.     

基于高斯过程分类器的三维模型多粒度语义检索

为解决三维模型语义检索中用户检索意图不一致问题,建立多粒度语义检索框架,使学习模型能够有效地适应用户的不同检索意图。首先对模型分类知识进行层次划分,形成语义概念的多粒度结构。然后提取一种多视图特征来描述三维模型的形状特性,并采用高斯过程分类器建立不同粒度层次上的学习模型,实现低层特征和查询概念之间的语义一致性描述。和已有研究相比,多粒度语义检索框架使用户可通过语义粒度级别变化进行检索意图设置,从而检索结果尽可能符合用户语义。在实验部分,采用三维模型基准数据库对框架进行算法性能测试。结果表明,检索准确率要明显提高,并且符合人类思维特点。     

Convolutional deep learning for 3D object retrieval

In recent years, with the development of 3D technologies, 3D model retrieval has become a hot topic. The key point of 3D model retrieval is to extract robust feature for 3D model representation. In order to improve the effectiveness of method on 3D model retrieval, this paper proposes a feature extraction model based on convolutional neural networks (CNN). First, we extract a set of 2D images from 3D model to represent each 3D object. SIFT detector is utilized to detect interesting points from each 2D image and extract interesting patches to represent local information of each 3D model. X-means is leveraged to generate the CNN filters. Second, a single CNN layer learns low-level features which are then given as inputs to multiple recursive neural networks (RNN) in order to compose higher order features. RNNs can generate the final feature for 2D image representation. Finally, nearest neighbor is used to compute the similarity between different 3D models in order to handle the retrieval problem. Extensive comparison experiments were on the popular ETH and MV-RED 3D model datasets. The results demonstrate the superiority of the proposed method.     

3D model retrieval via single image based on feature mapping

With the development of manufacture, more and more 3D models are generated by users and many differnet factories. 3D model retrieval has been receiving more and more attention in computer vision and the field of data analysis. In this paper, we propose a novel 3D model retrieval algorithm by cross-modal feature mapping (CMFM), which utilize one single image as query information to address 3D model retrieval problem. Specifically, in this paper, we first proposed to leverage 2D image to handle 3d model retrieval problem, which is one new problem in this field. The proposed feature learning method can benefit: 1) avoiding the interference of query image recorded by different visual sensor; 2) handling cross-modal data retrieval by simple computer vision technologies, which can guarantee the performance of retrieval and also control that the retrieval time hold a low level; 3) the low complexity of this method can guarantee that this method can be applied in many fields. Finally, we validate the retrieval method on three popular datasets. Extensive comparison experiments show the superiority of the proposed mehtod. To the best of our knowledge, it is the first method to handle 3D model retreival based on one single 2D image.     

宽基线主动视觉中感兴趣目标的对应技术

在主动视觉系统中,通常需要多个代理对同一场景中的感兴趣目标进行协同处理,以提高系统智能分析感兴趣目标的能力。其中,基于多视几何关系解决感兴趣目标的对应问题是协同处理的基础。一方面,主动视觉系统一般工作在宽基线条件下,这增加了对应问题描述的复杂性;另一方面,主动视觉系统以最佳视角观察目标,因此摄像头需做实时的姿态调整,由此导致的视间几何关系变化进一步加深了对应问题的解决难度。本文基于仿射不变的几何特征,建立宽基线条件下的多视几何关系,并针对频繁使用几何特征不能满足主动视觉系统实时要求的问题,提出一种快速更新多视几何关系的方法,并在多视几何约束下实现对应感兴趣目标的鲁棒标识。实验结果表明,该方法能解决宽基线主动视觉系统中感兴趣目标的复杂对应问题,并能达到实时要求。     

基于半监督典型相关分析的多视图维数约简

为了有效地在半监督多视图情景下进行维数约简,提出了使用非负低秩图进行标签传播的半监督典型相关分析方法。非负低秩图捕获的全局线性近邻可以利用直接邻居和间接可达邻居的信息维持全局簇结构,同时,低秩的性质可以保持图的压缩表示。当无标签样本通过标签传播算法获得估计的标签信息后,在每个视图上构建软标签矩阵和概率类内散度矩阵。然后,通过最大化不同视图同类样本间相关性的同时最小化每个视图低维特征空间类内变化来提升特征鉴别能力。实验表明所提方法比已有相关方法能够取得更好的识别性能且更鲁棒。     

Sketch-based 3D model retrieval by incorporating 2D-3D alignment

Sketch-based 3D model retrieval is very important for applications such as 3D modeling and recognition. In this paper, a sketch-based retrieval algorithm is proposed based on a 3D model feature named View Context and 2D relative shape context matching. To enhance the accuracy of 2D sketch-3D model correspondence as well as the retrieval performance, we propose to align a 3D model with a query 2D sketch before measuring their distance. First, we efficiently select some candidate views from a set of densely sampled views of the 3D model to align the sketch and the model based on their View Context similarities. Then, we compute the more accurate relative shape context distance between the sketch and every candidate view, and regard the minimum one as the sketch-model distance. To speed up retrieval, we precompute the View Context and relative shape context features of the sample views of all the 3D models in the database. Comparative and evaluative experiments based on hand-drawn and standard line drawing sketches demonstrate the effectiveness and robustness of our approach and it significantly outperforms several latest sketch-based retrieval algorithms.     

基于多视图网络三维形状检索的通用扰动攻击

几何深度学习模型在三维形状检索任务中已应用,其安全评估工作也引起了研究者们的关注.该文针对三维形状检索评估提出一种基于多视图通用扰动攻击(MvUPA)的对抗攻击方法,其具有高成功率的攻击效果.首先设计多视角深度全景图检索模型,训练适用于视图类三维形状检索的高效嵌入向量;其次,为三维形状检索提出有益于通用扰动更新的损失函...     

融合信息熵和CNN 的基于手绘的三维模型检索

基于手绘草图的三维模型检索(SBSR)已成为三维模型检索、模式识别与计算机视 觉领域的一个研究热点。与传统方法相比,基于卷积神经网络(CNN)的三维深度表示方法在三 维模型检索任务中性能优势非常明显。本文提出了一种基于手绘图像融合信息熵和CNN 的三 维模型检索方法。首先,通过计算模型投影图的信息熵得到模型的代表性视图,并将代表性视 图经过边缘检测等处理得到三维模型投影图的轮廓图像;然后,将轮廓图像和手绘草图输入到 CNN 中提取特征描述子,并进行特征匹配。本文方法在Shape Retrieval Contest (SHREC) 2012 数据库和SHREC 2013 数据库上进行实验。实验证明,该方法的效果较其他传统方法检索准确 度更高。