大规模图神经网络系统综述

图神经网络(GNN)是一类基于深度学习的处理图域信息的方法,它通过将图广播操作和深度学习算法结合,可以让图的结构信息和顶点属性信息都参与到学习中,在顶点分类、图分类、链接预测等应用中表现出良好的效果和可解释性,已成为一种广泛应用的图分析方法.然而现有主流的深度学习框架(如TensorFlow、PyTorch等)没有为图...     

图知识蒸馏综述：算法分类与应用分析

图数据, 如引文网络, 社交网络和交通网络, 广泛地存在现实生活中. 图神经网络凭借强大的表现力受到广泛关注, 在各种各样的图分析应用中表现卓越. 然而, 图神经网络的卓越性能得益于标签数据和复杂的网络模型, 而标签数据获取困难且计算资源代价高昂. 为了解决数据标签的稀疏性和模型计算的高复杂性问题, 知识蒸馏被引入到图神经网络中. 知识蒸馏是一种利用性能更好的大模型(教师模型)的软标签监督信息来训练构建的小模型(学生模型), 以期达到更好的性能和精度. 因此, 如何面向图数据应用知识蒸馏技术成为重大研究挑战, 但目前尚缺乏对于图知识蒸馏研究的综述. 旨在对面向图的知识蒸馏进行全面综述, 首次系统地梳理现有工作, 弥补该领域缺乏综述的空白. 具体而言, 首先介绍图和知识蒸馏背景知识; 然后, 全面梳理3类图知识蒸馏方法, 面向深度神经网络的图知识蒸馏、面向图神经网络的图知识蒸馏和基于图知识的模型自蒸馏方法, 并对每类方法进一步划分为基于输出层、基于中间层和基于构造图知识方法; 随后, 分析比较各类图知识蒸馏算法的设计思路, 结合实验结果总结各类算法的优缺点; 此外, 还列举图知识蒸馏在计算机视觉、自然语言处理、推荐系统等领域的应用; 最后对图知识蒸馏的发展进行总结和展望. 还将整理的图知识蒸馏相关文献公开在GitHub平台上, 具体参见: https://github.com/liujing1023/Graph-based-Knowledge-Distillation.     

Autonomous Learning System Towards Mobile Intelligence

How to efficiently deploy machine learning models on mobile devices has drawn a lot of attention in both academia and industries, among which the model training is a critical part. However, with increasingly public attention on data privacy and the recently adopted laws and regulations, it becomes harder for developers to collect training data from users and thus they cannot train high-quality models. Researchers have been exploring approaches to training neural networks on decentralized data. Those efforts will be summarized and their limitations be pointed out. To this end, this work presents a novel neural network training paradigm on mobile devices, which distributes all training computations associated with private data on local devices and requires no data to be uploaded in any form. Such training paradigm is named autonomous learning. To deal with two main challenges of autonomous learning, i.e., limited data volume and insufficient computing power available on mobile devices, this paper designs and implements the first autonomous learning system AutLearn. It incorporates the cloud (public data and pre-training)--client (private data and transfer learning) cooperation methodology and data augmentation techniques to ensure the model convergence on mobile devices. Furthermore, by optimization techniques such as model compression, neural network compiler, and runtime cache reuse, AutLearn can significantly reduce the on-client training cost. Two classical scenarios of autonomous learning are implemented based on AutLearn,with a set of experiments carried out. The results show that AutLearn can train the neural networks with comparable or even higher accuracy compared to traditional centralized/federated training mode with privacy preserved. AutLearn can also remarkably cut the computational and energy cost of neural network training on mobile devices.     

图神经网络在复杂图挖掘上的研究进展

图神经网络对非欧式空间数据建立了深度学习框架,相比传统网络表示学习模型,它对图结构能够实施更加深层的信息聚合操作.近年来,图神经网络完成了向复杂图结构的迁移,诞生了一系列基于复杂图的图神经网络模型.然而,现有综述文章缺乏对复杂图神经网络全面、系统的归纳和总结工作.将复杂图分为异质图、动态图和超图3种类型.将异质图神经网络按照信息聚合方式划分为关系类型感知和元路径感知两大类,在此基础上,分别介绍普通异质图和知识图谱.将动态图神经网络按照处理时序信息的方式划分成基于循环神经网络、基于自编码器以及时空图神经网络三大类.将超图神经网络按照是否将超图展开成成对图划分为展开型和非展开型两大类,进一步按照展开方式将展开型划分成星形展开、团式展开和线形展开3种类型.详细阐述了每种算法的核心思想,比较了不同算法间的优缺点,系统列举了各类复杂图神经网络的关键算法、(交叉)应用领域和常用数据集,并对未来可能的研究方向进行了展望.     

面向多GPU的图神经网络训练加速

图神经网络由于其强大的表示能力和灵活性最近取得了广泛的关注. 随着图数据规模的增长和显存容量的限制, 基于传统的通用深度学习系统进行图神经网络训练已经难以满足要求, 无法充分发挥GPU设备的性能. 如何高效利用GPU硬件进行图神经网络的训练已经成为该领域重要的研究问题之一. 传统做法是基于稀疏矩阵乘法, 完成图神经网络中的计算过程, 当面对GPU显存容量限制时, 通过分布式矩阵乘法, 把计算任务分发到每个设备上, 这类方法的主要不足有: (1)稀疏矩阵乘法忽视了图数据本身的稀疏分布特性, 计算效率不高; (2)忽视了GPU本身的计算和访存特性, 无法充分利用GPU硬件. 为了提高训练效率, 现有一些研究通过图采样方法, 减少每轮迭代的计算带价和存储需求, 同时也可以支持灵活的分布式拓展, 但是由于采样随机性和方差, 它们往往会影响训练的模型精度. 为此, 提出了一套面向多GPU的高性能图神经网络训练框架, 为了保证模型精度, 基于全量图进行训练, 探索了不同的多GPU图神经网络切分方案, 研究了GPU上不同的图数据排布对图神经网络计算过程中GPU性能的影响, 并提出了稀疏块感知的GPU访存优化技术. 基于C++和CuDNN实现了该原型系统, 在4个不同的大规模GNN数据集上的实验表明: (1)通过图重排优化, 提高了GPU约40%的缓存命中率, 计算加速比可达2倍; (2)相比于现有系统DGL, 取得了5.8倍的整体加速比.     

V2G网络中基于联邦学习和CNN-BiLSTM的DDoS攻击检测

DDoS攻击是V2G网络的重要威胁之一,它可以在短时间内耗尽服务器的通信资源。 此前方法以集中式模型为主,将数据从边缘设备传输到中央服务器进行训练可能会将数据暴露给各种攻击。研究了一种基于联邦学习的入侵检测系统,首先,考虑到V2G网络数据的高维性和数据间的时间依赖性,将采集的数据通过改进的特征选择算法进行降维,减少冗余特征,再将处理后的数据传入到融合了卷积神经网络和双向长短时记忆网络的混合模型中,捕获数据中的时间依赖性,并引入批标准化防止神经网络训练过程中出现梯度消失问题; 其次,为了防止隐私泄露,结合联邦学习的固有特性,允许数据留在本地用于神经网络模型的训练;为了解决联邦学习通信造成网络负载压力过大的问题,设计了一种通过设置动态通信阈值筛选参与更新最优边缘设备的方案以减轻网络负载压力。实验结果表明,该方法的准确率可以高达99.95%,单轮通信时间减少了1.7 s。     

Multivariate time series prediction of complex systems based on graph neural networks with location embedding graph structure learning

Graph convolutional neural networks (GNNs) have an excellent expression ability for complex systems. However, the smoothing hypothesis based GNNs have certain limitations for complex process industrial systems with high dynamics and noisy environment. In addition, it is difficult to obtain an accurate information about the interconnections of sensor networks in manufacturing systems, which brings challenges to the application of GNNs. This paper introduces a graph convolution filter with a serial alternating structure of low-pass filter and high-pass filter to alleviate the problem of node feature loss. Furthermore, we propose a simple and effective method to learn graph structure information during training. This method combines the advantages of graph structure learning based on metric method and direct optimization method. Finally, a spatiotemporal parallel feature extraction framework for multivariate time series prediction is constructed. Experiments are carried out on real industrial datasets, and the results demonstrate the effectiveness of the model.     

基于强化联邦图神经网络的个性化公共安全突发事件检测方法

近年来,将公共安全数据转换为图的形式,通过图神经网络构造节点表示应用于下游任务的方法,充分利用了公共安全数据的实体与关联信息,取得了较好的效果.为了提高模型的有效性,需要大量的高质量数据,但是高质量的数据通常归属于政府、公司和组织,很难通过数据集中的方式使模型学习到有效的事件检测模型.由于各数据拥有方的关注主题与收集时间不同,数据之间存在Non-IID的问题.传统的假设一个全局模型可以适合所有客户端的方法难以解决此类问题.本文提出了基于强化联邦图神经网络的公共安全突发事件检测方法PPSED,各客户端采用多方协作的方式训练个性化的模型来解决本地的突发事件检测任务.设计联邦公共安全突发事件检测模型的本地训练与梯度量化模块,采用基于图采样的minibatch机制的GraphSage构造公共安全突发事件检测本地模型,以减小数据Non-IID的影响,采用梯度量化方法减小梯度通信的消耗.设计基于随机图嵌入的客户端状态感知模块,在保护隐私的同时更好地保留客户端模型有价值的梯度信息.设计强化联邦图神经网络的个性化梯度聚合与量化策略,采用DDPG拟合个性化联邦学习梯度聚合加权策略,并根据权重决定是否对梯度进行量化,对模型的性能与通信压力进行平衡.通过在微博平台收集的公共安全数据集和三个公开的图数据集进行了大量的实验,实验结果表明了提出的方法的有效性.     

基于非鲁棒特征的图卷积神经网络对抗训练方法

图卷积神经网络可以通过图卷积提取图数据的有效信息,但容易受到对抗攻击的影响导致模型性能下降。对抗训练能够用于提升神经网络鲁棒性,但由于图的结构及节点特征通常是离散的,无法直接基于梯度构造对抗扰动,而在模型的嵌入空间中提取图数据的特征作为对抗训练的样本,能够降低构造复杂度。借鉴集成学习思想,提出一种基于非鲁棒特征的图卷积神经网络对抗训练方法VDERG,分别针对拓扑结构和节点属性两类特征,构建两个图卷积神经网络子模型,通过嵌入空间提取非鲁棒特征,并基于非鲁棒特征完成对抗训练,最后集成两个子模型输出的嵌入向量作为模型节点表示。实验结果表明,提出的对抗训练方法在干净数据上的准确率平均提升了0.8%,在对抗攻击下最多提升了6.91%的准确率。     

基于信誉评估机制和区块链的移动网络联邦学习方案

联邦学习是一种新兴的分布式机器学习技术,通过将训练任务下放到用户端,仅将训练得到的模型参数发送给服务端,整个过程并不需要参与方直接共享数据,从而很大限度上规避了隐私问题。然而,这种学习模式中移动用户间没有预先建立信任关系,用户之间进行合作训练时会存在安全隐患。针对上述问题,提出一种基于信誉评估机制和区块链的移动网络联邦学习方案,该方案允许服务端利用主观逻辑模型对参与训练的移动用户进行信誉评估,并且基于区块链智能合约技术为其提供可信的信誉意见共享环境和动态访问策略接口。理论和实验分析结果表明,此方案可以使服务端选择可靠的用户进行训练,同时能够实现更公平和有效的信誉计算,提高联邦学习的准确性。     

Embedded federated learning over a LoRa mesh network

In on-device training of machine learning models on microcontrollers a neural network is trained on the device. A specific approach for collaborative on-device training is federated learning. In this paper, we propose embedded federated learning on microcontroller boards using the communication capacity of a LoRa mesh network. We apply a dual board design: The machine learning application that contains a neural network is trained for a keyword spotting task on the Arduino Portenta H7. For the networking of the federated learning process, the Portenta is connected to a TTGO LORA32 board that operates as a router within a LoRa mesh network. We experiment the federated learning application on the LoRa mesh network and analyze the network, system, and application level performance. The results from our experimentation suggest the feasibility of the proposed system and exemplify an implementation of a distributed application with re-trainable compute nodes, interconnected over LoRa, entirely deployed at the tiny edge.     

图自编码器推荐研究综述

图自编码器GAE是一种源自图神经网络的学习框架,在编码器中引入聚合邻域节点的思想,解码器对图结构数据进行解码,重构图结构数据;在模型中引入监督模块,可以提高图结构数据在模型中的嵌入完整性和数据生成的准确性;编解码可以采用不同的神经网络,从而利用不同神经网络的优点。近年来GAE推荐逐渐成为推荐系统研究的热点。从无监督学习与半监督学习方面分析了已有GAE推荐研究取得的进展;探讨了已有GAE推荐模型存在用户冷启动问题、可解释性差、模型复杂度高和难以处理数据的多源异构性等方面的问题;并从跨领域推荐,结合传统推荐方法,引入注意力机制,融合各类场景等研究方向对未来GAE推荐进行展望。     

Towards Defense Against Adversarial Attacks on Graph Neural Networks via Calibrated Co-Training

Graph neural networks (GNNs) have achieved significant success in graph representation learning. Nevertheless, the recent work indicates that current GNNs are vulnerable to adversarial perturbations, in particular structural perturbations. This, therefore, narrows the application of GNN models in real-world scenarios. Such vulnerability can be attributed to the model’s excessive reliance on incomplete data views (e.g., graph convolutional networks (GCNs) heavily rely on graph structures to make predictions). By integrating the information from multiple perspectives, this problem can be effectively addressed, and typical views of graphs include the node feature view and the graph structure view. In this paper, we proposeC²oG, which combines these two typical views to train sub-models and fuses their knowledge through co-training. Due to the orthogonality of the views, sub-models in the feature view tend to be robust against the perturbations targeted at sub-models in the structure view.C²oGallows sub-models to correct one another mutually and thus enhance the robustness of their ensembles. In our evaluations,C²oGsignificantly improves the robustness of graph models against adversarial attacks without sacrificing their performance on clean datasets.     

Pre-Train and Learn:Preserving Global Information for Graph Neural Networks

Graph neural networks(GNNs) have shown great power in learning on graphs.However,it is still a challenge for GNNs to model information faraway from the source node.The ability to preserve global information can enhance graph representation and hence improve classification precision.In the paper,we propose a new learning framework named G-GNN(Global information for GNN) to address the challenge.First,the global structure and global attribute features of each node are obtained via unsupervised pre-training,and those global features preserve the global information associated with the node.Then,using the pre-trained global features and the raw attributes of the graph,a set of parallel kernel GNNs is used to learn different aspects from these heterogeneous features.Any general GNN can be used as a kernal and easily obtain the ability of preserving global information,without having to alter their own algorithms.Extensive experiments have shown that state-of-the-art models,e.g.,GCN,GAT,Graphsage and APPNP,can achieve improvement with G-GNN on three standard evaluation datasets.Specially,we establish new benchmark precision records on Cora(84.31％) and Pubmed(80.95％) when learning on attributed graphs.     

Multi-class pattern classification using neural networks

Multi-class pattern classification has many applications including text document classification, speech recognition, object recognition, etc. Multi-class pattern classification using neural networks is not a trivial extension from two-class neural networks. This paper presents a comprehensive and competitive study in multi-class neural learning with focuses on issues including neural network architecture, encoding schemes, training methodology and training time complexity. Our study includes multi-class pattern classification using either a system of multiple neural networks or a single neural network, and modeling pattern classes using one-against-all, one-against-one, one-against-higher-order, and P-against-Q. We also discuss implementations of these approaches and analyze training time complexity associated with each approach. We evaluate six different neural network system architectures for multi-class pattern classification along the dimensions of imbalanced data, large number of pattern classes, large vs. small training data through experiments conducted on well-known benchmark data.     

Round efficient privacy-preserving federated learning based on MKFHE

Federated learning (FL) was created with the intention of enabling collaborative training of models without the need for direct data exchange. However, data leakage remains an issue in FL. Multi-Key Fully Homomorphic Encryption (MKFHE) is a promising technique that allows computations on ciphertexts encrypted by different parties. MKFHE’s aptitude to handle multi-party data makes it an ideal tool for implementing privacy-preserving federated learning.We present a multi-hop MKFHE with compact ciphertext. MKFHE allows computations on data encrypted by different parties. In MKFHE, the compact ciphertext means that the size of the ciphertext is independent of the number of parties. The multi-hop property means that parties can dynamically join the homomorphic computation at any time. Prior MKFHE schemes were limited by their inability to combine these desirable properties. To address this limitation, we propose a multi-hop MKFHE scheme with compact ciphertext based on the random sample common reference string(CRS). We construct our scheme based on the residue number system (RNS) variant CKKS17 scheme, which enables efficient homomorphic computation over complex numbers due to the RNS representations of numbers.We construct a round efficient privacy-preserving federated learning based on our multi-hop MKFHE. In FL, there is always the possibility that some clients may drop out during the computation. Previous HE-based FL methods did not address this issue. However, our approach takes advantage of multi-hop MKFHE that users can join dynamically and constructs an efficient federated learning scheme that reduces interactions between parties. Compared to other HE-based methods, our approach reduces the number of interactions during a round from 3 to 2. Furthermore, in situations where some users fail, we are able to reduce the number of interactions from 3 to just 1.     

一种卷积神经网络和极限学习机相结合的人脸识别方法

卷积神经网络是一种很好的特征提取器,但却不是最佳的分类器,而极限学习机能够很好地进行分类,却不能学习复杂的特征,根据这两者的优点和缺点,将它们结合起来,提出一种新的人脸识别方法。卷积神经网络提取人脸特征,极限学习机根据这些特征进行识别。本文还提出固定卷积神经网络的部分卷积核以减少训练参 数,从而提高识别精度的方法。在人脸库ORL和XM2VTS上进行测试的结果表明,本文的结合方法能有效提高人脸识别的识别率,而且固定部分卷积核的方式在训练样本少时具有优势。     

A comparison of statistical relational learning and graph neural networks for aggregate graph queries

Statistical relational learning (SRL) and graph neural networks (GNNs) are two powerful approaches for learning and inference over graphs. Typically, they are evaluated in terms of simple metrics such as accuracy over individual node labels. Complex aggregate graph queries (AGQ) involving multiple nodes, edges, and labels are common in the graph mining community and are used to estimate important network properties such as social cohesion and influence. While graph mining algorithms support AGQs, they typically do not take into account uncertainty, or when they do, make simplifying assumptions and do not build full probabilistic models. In this paper, we examine the performance of SRL and GNNs on AGQs over graphs with partially observed node labels. We show that, not surprisingly, inferring the unobserved node labels as a first step and then evaluating the queries on the fully observed graph can lead to sub-optimal estimates, and that a better approach is to compute these queries as an expectation under the joint distribution. We propose a sampling framework to tractably compute the expected values of AGQs. Motivated by the analysis of subgroup cohesion in social networks, we propose a suite of AGQs that estimate the community structure in graphs. In our empirical evaluation, we show that by estimating these queries as an expectation, SRL-based approaches yield up to a 50-fold reduction in average error when compared to existing GNN-based approaches.

     

基于差分隐私的分段裁剪联邦学习算法

为解决现有的差分隐私联邦学习算法中使用固定的裁剪阈值和噪声尺度进行训练,从而导致数据隐私泄露、模型精度较低的问题,提出了一种基于差分隐私的分段裁剪联邦学习算法。首先,根据客户端的隐私需求分为隐私需求高和低。对于高隐私需求用户使用自适应裁剪来动态裁剪梯度;而低隐私需求用户则采用比例裁剪。其次根据裁剪后阈值大小自适应地添加噪声尺度。通过实验分析可得,该算法可以更好地保护隐私数据,同时通信代价也低于ADP-FL和DP-FL算法,并且与ADP-FL和DP-FL相比,模型准确率分别提高了2.25%和4.41%。     

一种面向图神经网络的图重构防御方法

近年来,图神经网络在图表示学习领域中取得了较好表现广泛应用于日常生活中,例如电子商务、社交媒体和生物学等.但是研究表明,图神经网络容易受到精心设计的对抗攻击迷惑,使其无法正常工作.因此,提高图神经网络的鲁棒性至关重要.已有研究提出了一些提高图神经网络鲁棒性的防御方法,然而如何在确保模型主任务性能的前提下降低对抗攻击的攻击成功率仍存在挑战.通过观察不同攻击产生的对抗样本发现,对抗攻击生成的对抗连边所对应的节点对之间通常存在低结构相似性和低节点特征相似性的特点.基于上述发现,提出了一种面向图神经网络的图重构防御方法GRD-GNN,分别从图结构和节点特征考虑,采用共同邻居数和节点相似度2种相似度指标检测对抗连边并实现图重构,使得重构的图结构删除对抗连边,且添加了增强图结构关键特征的连边,从而实现有效防御.最后,论文在3个真实数据集上展开防御实验,验证了GRD-GNN相比其他防御方法均能取得最佳的防御性能,且不影响正常图数据的分类任务.此外,利用可视化方法对防御结果做解释,解析方法的有效性.