Multi-Modal Domain Adaptation Variational Autoencoder for EEG-Based Emotion Recognition

Traditional electroencephalograph (EEG)-based emotion recognition requires a large number of calibration samples to build a model for a specific subject, which restricts the application of the affective brain computer interface (BCI) in practice. We attempt to use the multi-modal data from the past session to realize emotion recognition in the case of a small amount of calibration samples. To solve this problem, we propose a multi-modal domain adaptive variational autoencoder (MMDA-VAE) method, which learns shared cross-domain latent representations of the multi-modal data. Our method builds a multi-modal variational autoencoder (MVAE) to project the data of multiple modalities into a common space. Through adversarial learning and cycle-consistency regularization, our method can reduce the distribution difference of each domain on the shared latent representation layer and realize the transfer of knowledge. Extensive experiments are conducted on two public datasets, SEED and SEED-IV, and the results show the superiority of our proposed method. Our work can effectively improve the performance of emotion recognition with a small amount of labelled multi-modal data.      

面向不完整多视图聚类的深度互信息最大化方法

多视图聚类是无监督学习领域研究热点之一,近年来涌现出许多优秀的多视图聚类工作,但其中大多数方法均假设各视图是完整的,然而真实场景下数据收集过程极容易发生缺失,造成部分视图不完整。同时,很多方法采取传统机器学习方法（即浅层模型）对数据进行特征学习,这导致模型难以挖掘高维数据内的复杂信息。针对以上问题,本文提出一种面向不完整多视图聚类的深度互信息最大化方法。首先利用深度自编码器挖掘各视图深层次的隐含特征,并通过最大化潜在表示间的互信息来学习各视图间的一致性知识。然后,对于不完整视图中的缺失数据,利用多视图的公共潜在表示进行补全。此外,本文采用一种自步学习策略对网络进行微调,从易到难地学习数据集中的样本,得到更加宜于聚类的特征表示。最后,在多个真实数据集上进行实验,验证了本文方法的有效性。     

Generating Adversarial Samples on Multivariate Time Series using Variational Autoencoders

Classification models for multivariate time series have drawn the interest of many researchers to the field with the objective of developing accurate and efficient models. However, limited research has been conducted on generating adversarial samples for multivariate time series classification models. Adversarial samples could become a security concern in systems with complex sets of sensors. This study proposes extending the existing gradient adversarial transformation network (GATN) in combination with adversarial autoencoders to attack multivariate time series classification models. The proposed model attacks classification models by utilizing a distilled model to imitate the output of the multivariate time series classification model. In addition, the adversarial generator function is replaced with a variational autoencoder to enhance the adversarial samples. The developed methodology is tested on two multivariate time series classification models: 1-nearest neighbor dynamic time warping (1-NN DTW) and a fully convolutional network (FCN). This study utilizes 30 multivariate time series benchmarks provided by the University of East Anglia (UEA) and University of California Riverside (UCR). The use of adversarial autoencoders shows an increase in the fraction of successful adversaries generated on multivariate time series. To the best of our knowledge, this is the first study to explore adversarial attacks on multivariate time series. Additionally, we recommend future research utilizing the generated latent space from the variational autoencoders.      

Adversarial Heterogeneous Network Embedding with Metapath Attention Mechanism

Heterogeneous information network (HIN)-structured data provide an effective model for practical purposes in real world. Network embedding is fundamental for supporting the network-based analysis and prediction tasks. Methods of network embedding that are currently popular normally fail to effectively preserve the semantics of HIN. In this study, we propose AGA2Vec, a generative adversarial model for HIN embedding that uses attention mechanisms and meta-paths. To capture the semantic information from multi-typed entities and relations in HIN, we develop a weighted meta-path strategy to preserve the proximity of HIN. We then use an autoencoder and a generative adversarial model to obtain robust representations of HIN. The results of experiments on several real-world datasets show that the proposed approach outperforms state-of-the-art approaches for HIN embedding.

     

Joint user profiling with hierarchical attention networks

User profiling by inferring user personality traits, such as age and gender, plays an increasingly important role in many real-world applications. Most existing methods for user profiling either use only one type of data or ignore handling the noisy information of data. Moreover, they usually consider this problem from only one perspective. In this paper, we propose a joint user profiling model with hierarchical attention networks (JUHA) to learn informative user representations for user profiling. Our JUHA method does user profiling based on both inner-user and inter-user features. We explore inner-user features from user behaviors (e.g., purchased items and posted blogs), and inter-user features from a user-user graph (where similar users could be connected to each other). JUHA learns basic sentence and bag representations from multiple separate sources of data (user behaviors) as the first round of data preparation. In this module, convolutional neural networks (CNNs) are introduced to capture word and sentence features of age and gender while the self-attention mechanism is exploited to weaken the noisy data. Following this, we build another bag which contains a user-user graph. Inter-user features are learned from this bag using propagation information between linked users in the graph. To acquire more robust data, inter-user features and other inner-user bag representations are joined into each sentence in the current bag to learn the final bag representation. Subsequently, all of the bag representations are integrated to lean comprehensive user representation by the self-attention mechanism. Our experimental results demonstrate that our approach outperforms several state-of-the-art methods and improves prediction performance.     

基于元学习的多视图对比融合冷启动推荐算法

针对当前冷启动推荐模型在处理异质信息网络时难以充分挖掘结构与语义信息,以及忽略网络中用户行为属性的问题,提出了一种基于元学习的多视图对比融合冷启动推荐算法（MVC-ML）。该算法在模型层和数据层双重作用下,有效缓解了冷启动问题。在MVC-ML算法框架中,首先通过元路径视图提取异质信息网络的高阶语义信息;其次,利用网络模式视图捕获网络的结构特征;再接着,通过聚类视图分析用户行为属性信息;最后,运用对比学习方法,将上述三个视图中提炼的信息进行综合融合,以生成准确的表示向量。通过在DBook等三个数据集上的实验验证,MVC-ML模型在冷启动场景下相较MetaHIN等传统异质信息网络模型,在MAE上降低了1.67%,在RMSE上降低了2.06%,同时nDCG@K提高了1.48%。这些数据充分证实了MVC-ML算法的合理性和有效性。     

3D shape recognition based on multi-modal information fusion

The classification and retrieval of 3D models have been widely used in the field of multimedia and computer vision. With the rapid development of computer graphics, different algorithms corresponding to different representations of 3D models have achieved the best performance. The advances in deep learning also encourage various deep models for 3D feature representation. For multi-view, point cloud, and PANORAMA-view, different models have shown significant performance on 3D shape classification. However, There’s not a way to consider utilizing the fusion information of multi-modal for 3D shape classification. In our opinion, We propose a novel multi-modal information fusion method for 3D shape classification, which can fully utilize the advantage of different modal to predict the label of class. More specifically, the proposed can effectively fuse more modal information. it is easy to utilize in other similar applications. We have evaluated our framework on the popular dataset ModelNet40 for the classification task on 3D shape. Series experimental results and comparisons with state-of-the-art methods demonstrate the validity of our approach.

     

融合全局信息的注意力增强会话推荐方法

近年来,基于会话推荐系统（session-based recommender system,SRS）的应用和研究是推荐系统的一个热门方向。如何利用用户会话信息进一步提升用户满意度和推荐精确度,是基于会话推荐系统的主要任务。目前大多数SBR模型仅基于目标会话对用户偏好建模,忽略了来自其他会话的物品转换信息,导致无法全面了解用户偏好。为了解决其局限性,提出融合全局上下文信息注意力增强的图神经网络模型（global context information graph neural networks for session-based recommendation,GCI-GNN）。该模型利用所有会话上的物品转换关系,更准确地获取用户偏好。具体而言,GCI-GNN从目标会话和全局会话学习物品向量表示。使用位置感知注意网络,将反向位置信息纳入物品嵌入中。考虑会话长度信息学习用户表示进而达到更有效的推荐。在Diginetica和Yoochoose数据集上进行实验,实验结果表明,相对最优的基准模型,GCI-GNN模型在Diginetica数据集各项指标上的提高超过2个百分点,在Yoochoose数据...     

Using shared representations to improve coordination and intent inference

In groupware, users must communicate about their intentions and aintain common knowledge via communication channels that are explicitly designed into the system. Depending upon the task, generic communication tools like chat or a shared whiteboard may not be sufficient to support effective coordination. We have previously reported on a methodology that helps the designer develop task specific communication tools, called coordinating representations, for groupware systems. Coordinating representations lend structure and persistence to coordinating information. We have shown that coordinating representations are readily adopted by a user population, reduce coordination errors, and improve performance in a domain task. As we show in this article, coordinating representations present a unique opportunity to acquire user information in collaborative, user-adapted systems. Because coordinating representations support the exchange of coordinating information, they offer a window onto task and coordination-specific knowledge that is shared by users. Because they add structure to communication, the information that passes through them can be easily exploited by adaptive technology. This approach provides a simple technique for acquiring user knowledge in collaborative, user-adapted systems. We document our application of this approach to an existing groupware system. Several empirical results are provided. First, we show how information that is made available by a coordinating representation can be used to infer user intentions. We also show how this information can be used to mine free text chat for intent information, and show that this information further enhances intent inference. Empirical data shows that an automatic plan generation component, which is driven by information from a coordinating representation, reduces coordination errors and cognitive effort for its users. Finally, our methodology is summarized, and we present a framework for comparing our approach to other strategies for user knowledge acquisition in adaptive systems.     

Multi-view Scene Flow Estimation: A View Centered Variational Approach

We present a novel method for recovering the 3D structure and scene flow from calibrated multi-view sequences. We propose a 3D point cloud parametrization of the 3D structure and scene flow that allows us to directly estimate the desired unknowns. A unified global energy functional is proposed to incorporate the information from the available sequences and simultaneously recover both depth and scene flow. The functional enforces multi-view geometric consistency and imposes brightness constancy and piecewise smoothness assumptions directly on the 3D unknowns. It inherently handles the challenges of discontinuities, occlusions, and large displacements. The main contribution of this work is the fusion of a 3D representation and an advanced variational framework that directly uses the available multi-view information. This formulation allows us to advantageously bind the 3D unknowns in time and space. Different from optical flow and disparity, the proposed method results in a nonlinear mapping between the images’ coordinates, thus giving rise to additional challenges in the optimization process. Our experiments on real and synthetic data demonstrate that the proposed method successfully recovers the 3D structure and scene flow despite the complicated nonconvex optimization problem.     

基于深度神经网络的字典学习

字典学习通常采用线性函数捕获数据潜在特征, 该方式无法充分提取数据的内在特征结构, 近年来深度学习方法因其强大的特征表示能力而备受关注, 由此本文提出一种结合深度学习与字典学习的非线性特征表示策略, 基于深度神经网络的字典学习(deep neural network-based dictionary learning, DNNDL). DNNDL将字典学习模块融入传统深度学习网络结构中, 在通过自编码器进行映射获取的低维嵌入空间中同时学习数据字典及在其上的稀疏表示系数, 从而实现端到端方式的数据潜在特征提取. DNNDL可为已有数据以及样本外点数据生成紧凑且具判别性的表示. DNNDL不仅是一种新的深度学习网络结构, 并且可将其看作为字典学习和深度学习相结合的统一框架. 通过在4个真实数据集上进行的大量实验, 验证表明所提方法较常用方法具有更好数据表示能力.     

基于深度自编码的多视图子空间聚类网络

传统子空间浅层聚类模型对于多视图和非线性数据的聚类性能不佳。为此,提出一种基于深度自编码器的多视图子空间聚类网络模型,通过在深度自编码器中引入子空间聚类中的“自我表示”特性以及加权稀疏表示,提升了多视图子空间聚类算法的学习能力。推导的深度自编码多视图子空间聚类算法能够聚类具有复杂结构的数据点。通过多视图数据集验证了提出算法的有效性。结果表明,该方法能够有效地挖掘数据固有的多样性聚类结构,并利用多个视图之间互补信息,在性能上与现有方法相比有较大的提升。     

Generating 3D Faces using Multi‐column Graph Convolutional Networks

In this work, we introduce multi‐column graph convolutional networks (MGCNs), a deep generative model for 3D mesh surfaces that effectively learns a non‐linear facial representation. We perform spectral decomposition of meshes and apply convolutions directly in the frequency domain. Our network architecture involves multiple columns of graph convolutional networks (GCNs), namely large GCN (L‐GCN), medium GCN (M‐GCN) and small GCN (S‐GCN), with different filter sizes to extract features at different scales. L‐GCN is more useful to extract large‐scale features, whereas S‐GCN is effective for extracting subtle and fine‐grained features, and M‐GCN captures information in between. Therefore, to obtain a high‐quality representation, we propose a selective fusion method that adaptively integrates these three kinds of information. Spatially non‐local relationships are also exploited through a self‐attention mechanism to further improve the representation ability in the latent vector space. Through extensive experiments, we demonstrate the superiority of our end‐to‐end framework in improving the accuracy of 3D face reconstruction. Moreover, with the help of variational inference, our model has excellent generating ability.     

Deep cross-view autoencoder network for multi-view learning

In many real-world applications, an increasing number of objects can be collected at varying viewpoints or by different sensors, which brings in the urgent demand for recognizing objects from distinct heterogeneous views. Although significant progress has been achieved recently, heterogeneous recognition (cross-view recognition) in multi-view learning is still challenging due to the complex correlations among views. Multi-view subspace learning is an effective solution, which attempts to obtain a common representation from downstream computations. Most previous methods are based on the idea of maximal correlation after feature extraction to establish the relationship among different views in a two-step manner, thus leading to performance deterioration. To overcome this drawback, in this paper, we propose a deep cross-view autoencoder network (DCVAE) that extracts the features of different views and establishes the correlation between views in one step to simultaneously handle view-specific, view-correlation, and consistency in a joint manner. Specifically, DCVAE contains self-reconstruction, newly designed cross-view reconstruction, and consistency constraint modules. Self-reconstruction ensures the view-specific, cross-view reconstruction transfers the information from one view to another view, and consistency constraint makes the representation of different views more consistent. The proposed model suffices to discover the complex correlation embedded in multi-view data and to integrate heterogeneous views into a latent common representation subspace. Furthermore, the 2D embeddings of the learned common representation subspace demonstrate the consistency constraint is valid and cross-view classification experiments verify the superior performance of DCVAE in the two-view scenario.

     

A novel multi-view learning developed from single-view patterns

The existing multi-view learning (MVL) learns how to process patterns with multiple information sources. In generalization this MVL is proven to have a significant advantage over the usual single-view learning (SVL). However, in most real-world cases we only have single source patterns to which the existing MVL is unable to be directly applied. This paper aims to develop a new MVL technique for single source patterns. To this end, we first reshape the original vector representation of single source patterns into multiple matrix representations. In doing so, we can change the original architecture of a given base classifier into different sub-ones. Each newly generated sub-classifier can classify the patterns represented with the matrix. Here each sub-classifier is taken as one view of the original base classifier. As a result, a set of sub-classifiers with different views are come into being. Then, one joint rather than separated learning process for the multi-view sub-classifiers is developed. In practice, the original base classifier employs the vector-pattern-oriented Ho–Kashyap classifier with regularization learning (called MHKS) as a paradigm which is not limited to MHKS. Thus, the proposed joint multi-view learning is named as MultiV-MHKS. Finally, the feasibility and effectiveness of the proposed MultiV-MHKS is demonstrated by the experimental results on benchmark data sets. More importantly, we have demonstrated that the proposed multi-view approach generally has a tighter generalization risk bound than its single-view one in terms of the Rademacher complexity analysis.     

基于融合元路径权重的异质网络表征学习

针对基于单条元路径的异质网络表征缺失异质信息网络中结构信息及其它元路径语义信息的问题,本文提出了基于融合元路径权重的异质网络表征学习方法.该方法对异质信息网络中元路径集合进行权重学习,进而对基于不同元路径的低维表征进行加权融合,得到融合不同元路径语义信息的异质网络表征.实验结果表明,基于融合元路径权重的异质网络表征学习具有良好的表征学习能力,可有效应用于数据挖掘.     

Mobile multi-view object image search

High user interaction capability of mobile devices can help improve the accuracy of mobile visual search systems. At query time, it is possible to capture multiple views of an object from different viewing angles and at different scales with the mobile device camera to obtain richer information about the object compared to a single view and hence return more accurate results. Motivated by this, we propose a new multi-view visual query model on multi-view object image databases for mobile visual search. Multi-view images of objects acquired by the mobile clients are processed and local features are sent to a server, which combines the query image representations with early/late fusion methods and returns the query results. We performed a comprehensive analysis of early and late fusion approaches using various similarity functions, on an existing single view and a new multi-view object image database. The experimental results show that multi-view search provides significantly better retrieval accuracy compared to traditional single view search.     

基于生成对抗网络的多视图学习与重构算法

同一事物通常需要从不同角度进行表达.然而,现实应用经常引出复杂的场景,导致完整视图数据很难获得.因此研究如何构建事物的完整视图具有重要意义.本文提出一种基于生成对抗网络（Generative adversarial networks,GAN）的多视图学习与重构算法,利用已知单一视图,通过生成式方法构建其他视图.为构建多视图通用的表征,提出新型表征学习算法,使得同一实例的任意视图都能映射至相同的表征向量,并保证其包含实例的重构信息.为构建给定事物的多种视图,提出基于生成对抗网络的重构算法,在生成模型中加入表征信息,保证了生成视图数据与源视图相匹配.所提出的算法的优势在于避免了不同视图间的直接映射,解决了训练数据视图不完整问题,以及构造视图与已知视图正确对应问题.在手写体数字数据集MNIST,街景数字数据集SVHN和人脸数据集CelebA上的模拟实验结果表明,所提出的算法具有很好的重构性能.     

基于双重低秩分解的不完整多视图子空间学习

多视图数据在现实世界中应用广泛,各种视角和不同的传感器有助于更好的数据表示,然而,来自不同视图的数据具有较大的差异,尤其当多视图数据不完整时,可能导致训练效果较差甚至失败。为了解决该问题,本文提出了一个基于双重低秩分解的不完整多视图子空间学习算法。所提算法通过两方面来解决不完整多视图问题:一方面,基于双重低秩分解子空间框架,引入潜在因子来挖掘多视图数据中缺失的信息;另一方面,通过预先学习的多视图数据低维特征获得更好的鲁棒性,并以有监督的方式来指导双重低秩分解。实验结果证明,所提算法较之前的多视图子空间学习算法有明显优势;即使对于不完整的多视图数据,该算法也具有良好的分类性能。     

Machine fault diagnosis with small sample based on variational information constrained generative adversarial network

In actual engineering scenarios, limited fault data leads to insufficient model training and over-fitting, which negatively affects the diagnostic performance of intelligent diagnostic models. To solve the problem, this paper proposes a variational information constrained generative adversarial network (VICGAN) for effective machine fault diagnosis. Firstly, by incorporating the encoder into the discriminator to map the deep features, an improved generative adversarial network with stronger data synthesis capability is established. Secondly, to promote the stable training of the model and guarantee better convergence, a variational information constraint technique is utilized, which constrains the input signals and deep features of the discriminator using the information bottleneck method. In addition, a representation matching module is added to impose restrictions on the generator, avoiding the mode collapse problem and boosting the sample diversity. Two rolling bearing datasets are utilized to verify the effectiveness and stability of the presented network, which demonstrates that the presented network has an admirable ability in processing fault diagnosis with few samples, and performs better than state-of-the-art approaches.