一种隐私保护的联邦学习框架

大数据时代,数据安全性和隐私性受到越来越多的关注和重视。联邦学习被视为是一种隐私保护的可行技术,允许从去中心化的数据中训练深度模型。针对电力投资系统中各部门因担心数据隐私信息泄露而带来的数据孤岛和隐私保护问题,提出了一种隐私保护的联邦学习框架,允许各部门自有数据在不出本地的情况下,联合训练模型。首先,提出了联邦学习的架构,支持分布式地训练模型;其次,引入同态加密技术,提出了隐私保护的联邦平均学习流程,在数据隐私保护的情况下,实现联合训练模型;最后,实验结果表明,该框架具有较好的收敛性,而且联合训练得到的模型具有较好的精度。     

梯度隐藏的安全聚类与隐私保护联邦学习

联邦学习是一种前沿的分布式机器学习算法,它在保障用户对数据控制权的同时实现了多方协同训练。然而,现有的联邦学习算法在处理Non-ⅡD数据、梯度信息泄露和动态用户离线等方面存在诸多问题。为了解决这些问题,基于四元数、零共享与秘密共享等技术,提出了一种梯度隐藏的安全聚类与隐私保护联邦学习。首先,借助四元数旋转技术隐藏首轮模型梯度,并且在确保梯度特征分布不变的情况下实现安全的聚类分层,从而解决Non-ⅡD数据导致的性能下降问题;其次,设计了一种链式零共享算法,采用单掩码策略保护用户模型梯度;然后,通过门限秘密共享来提升对用户离线情况的鲁棒性。与其他现有算法进行多维度比较表明,SCFL在Non-ⅡD数据分布下准确度提高3.13%~16.03%,整体运行时间提高3~6倍。同时,任何阶段均能保证信息传输的安全性,满足了精确性、安全性和高效性的设计目标。     

联邦学习中的隐私保护技术研究综述

近年来,联邦学习成为解决机器学习中数据孤岛与隐私泄露问题的新思路。联邦学习架构不需要多方共享数据资源,只要参与方在本地数据上训练局部模型,并周期性地将参数上传至服务器来更新全局模型,就可以获得在大规模全局数据上建立的机器学习模型。联邦学习架构具有数据隐私保护的特质,是未来大规模数据机器学习的新方案。然而,该架构的参数交互方式可能导致数据隐私泄露。目前,研究如何加强联邦学习架构中的隐私保护机制已经成为新的热点。从联邦学习中存在的隐私泄露问题出发,探讨了联邦学习中的攻击模型与敏感信息泄露途径,并重点综述了联邦学习中的几类隐私保护技术：以差分隐私为基础的隐私保护技术、以同态加密为基础的隐私保护技术、以安全多方计算（SMC）为基础的隐私保护技术。最后,探讨了联邦学习中隐私保护中的若干关键问题,并展望了未来研究方向。     

Round efficient privacy-preserving federated learning based on MKFHE

Federated learning (FL) was created with the intention of enabling collaborative training of models without the need for direct data exchange. However, data leakage remains an issue in FL. Multi-Key Fully Homomorphic Encryption (MKFHE) is a promising technique that allows computations on ciphertexts encrypted by different parties. MKFHE’s aptitude to handle multi-party data makes it an ideal tool for implementing privacy-preserving federated learning.We present a multi-hop MKFHE with compact ciphertext. MKFHE allows computations on data encrypted by different parties. In MKFHE, the compact ciphertext means that the size of the ciphertext is independent of the number of parties. The multi-hop property means that parties can dynamically join the homomorphic computation at any time. Prior MKFHE schemes were limited by their inability to combine these desirable properties. To address this limitation, we propose a multi-hop MKFHE scheme with compact ciphertext based on the random sample common reference string(CRS). We construct our scheme based on the residue number system (RNS) variant CKKS17 scheme, which enables efficient homomorphic computation over complex numbers due to the RNS representations of numbers.We construct a round efficient privacy-preserving federated learning based on our multi-hop MKFHE. In FL, there is always the possibility that some clients may drop out during the computation. Previous HE-based FL methods did not address this issue. However, our approach takes advantage of multi-hop MKFHE that users can join dynamically and constructs an efficient federated learning scheme that reduces interactions between parties. Compared to other HE-based methods, our approach reduces the number of interactions during a round from 3 to 2. Furthermore, in situations where some users fail, we are able to reduce the number of interactions from 3 to just 1.     

基于隐私保护联邦学习与区块链的图像分类方案

传统的中心化图像分类方法受制于数据隐私问题和计算资源限制,无法满足实际需求。现有的联邦学习框架依赖中心服务器,存在单点故障和数据中毒攻击等安全挑战。为解决这些问题,提出了一种面向隐私保护联邦学习与区块链的图像分类方案,通过将联邦学习与区块链技术相结合,实现在分布式环境下进行图像分类任务的可靠性和安全性。图像分类模型通过联邦学习进行训练,并上传至区块链网络进行验证和共识;在分类阶段,模型通过加权组合得到最终分类结果。实验结果表明,该方案在确保用户隐私的同时提高了图像分类的准确度,本方法为解决图像分类中的数据隐私和安全问题提供了一种有效途径,并为提高分类准确性作出了积极探索。     

Efficient asynchronous federated learning with sparsification and quantization

While data is distributed in multiple edge devices, federated learning (FL) is attracting more and more attention to collaboratively train a machine learning model without transferring raw data. FL generally exploits a parameter server and a large number of edge devices during the whole process of the model training, while several devices are selected in each round. However, straggler devices may slow down the training process or even make the system crash during training. Meanwhile, other idle edge devices remain unused. As the bandwidth between the devices and the server is relatively low, the communication of intermediate data becomes a bottleneck. In this article, we propose time-efficient asynchronous federated learning with sparsification and quantization, that is, TEASQ-Fed. TEASQ-Fed can fully exploit edge devices to asynchronously participate in the training process by actively applying for tasks. We utilize control parameters to choose an appropriate number of parallel edge devices, which simultaneously execute the training tasks. In addition, we introduce a caching mechanism and weighted averaging with respect to model staleness to further improve the accuracy. Furthermore, we propose a sparsification and quantitation approach to compress the intermediate data to accelerate the training. The experimental results reveal that TEASQ-Fed improves the accuracy (up to 16.67% higher) while accelerating the convergence of model training (up to twice faster).     

PPSS: A privacy-preserving secure framework using blockchain-enabled federated deep learning for Industrial IoTs

The growing reliance of industry 4.0/5.0 on emergent technologies has dramatically increased the scope of cyber threats and data privacy issues. Recently, federated learning (FL) based intrusion detection systems (IDS) promote the detection of large-scale cyber-attacks in resource-constrained and heterogeneous industrial systems without exposing data to privacy issues. However, the inherent characteristics of the latter have led to problems such as a trusted validation and consensus of the federation, unreliability, and privacy protection of model upload. To address these challenges, this paper proposes a novel privacy-preserving secure framework, named PPSS, based on the use of blockchain-enabled FL with improved privacy, verifiability, and transparency. The PPSS framework adopts the permissioned-blockchain system to secure multi-party computation as well as to incentivize cross-silo FL based on a lightweight and energy-efficient consensus protocol named Proof-of-Federated Deep-Learning (PoFDL). Specifically, we design two federated stages for global model aggregation. The first stage uses differentially private training of Stochastic Gradient Descent (DP-SGD) to enforce privacy protection of client updates, while the second stage uses PoFDL protocol to prove and add new model-containing blocks to the blockchain. We study the performance of the proposed PPSS framework using a new cyber security dataset (Edge-IIoT dataset) in terms of detection rate, precision, accuracy, computation, and energy cost. The results demonstrate that the PPSS framework system can detect industrial IIoT attacks with high classification performance under two distribution modes, namely, non-independent and identically distributed (Non-IID) and independent and identically distributed (IID).     

基于代理选举的高效异构联邦学习方法

物联网多样性终端设备在计算、存储、通信方面的异构性导致联邦学习效率不足。针对上述联邦训练过程中面临的问题,基于代理选举思路,提出了一种高效联邦学习算法。设计了基于马氏距离的代理节点选举策略,将设备的计算能力与闲置时长作为选举因素,选举性价比高的设备作为代理节点,充分发挥设备计算能力。进一步设计了基于代理节点的新型云边端联邦学习架构,提升了异构设备之间的联邦学习效率。基于MNIST和CIFAR-10公开数据集与智能家居设备真实数据的实验表明,该联邦学习方法的效率提高了22%。     

基于类别不平衡数据联邦学习的设备选择算法

考虑移动边缘计算下的联邦学习,其中全局服务器通过网络连接大量移动设备共同训练深度神经网络模型.全局类别不平衡和设备本地类别不平衡的数据分布往往会导致标准联邦平均算法性能下降.提出了一种基于组合式多臂老虎机在线学习算法框架的设备选择算法,并设计了一种类别估计方案.通过每一轮通信中选取与前次全局模型的类别测试性能偏移最互补的设备子集,使得训练后线性组合的全局模型各类别测试性能更平衡,从而获得更快的收敛性、更稳定的训练过程以及更好的测试性能.数值实验充分探究了不同参数对基于类别不平衡联邦平均算法的影响,以及验证了所提设备选择算法的有效性.     

Efficient privacy-preserving similar document detection

Similar document detection plays important roles in many applications, such as file management, copyright protection, plagiarism prevention, and duplicate submission detection. The state of the art protocols assume that the contents of files stored on a server (or multiple servers) are directly accessible. However, this makes such protocols unsuitable for any environment where the documents themselves are sensitive and cannot be openly read. Essentially, this assumption limits more practical applications, e.g., detecting plagiarized documents between two conferences, where submissions are confidential. We propose novel protocols to detect similar documents between two entities where documents cannot be openly shared with each other. The similarity measure used can be a simple cosine similarity on entire documents or on document fragments, enabling detection of partial copying. We conduct extensive experiments to show the practical value of the proposed protocols. While the proposed base protocols are much more efficient than the general secure multiparty computation based solutions, they are still slow for large document sets. We then investigate a clustering based approach that significantly reduces the running time and achieves over 90% of accuracy in our experiments. This makes secure similar document detection both practical and feasible.     

自适应异构联邦学习

目的 模型异构联邦学习由于允许参与者在不损害隐私的情况下独立设计其独特模型而受到越来越多的关注。现有的方法通常依赖于公共共享的相关数据或全局模型进行通信,极大地限制了适用性。且每个参与者的私有数据通常以不同的分布收集,导致数据异构问题。为了同时处理模型异构和数据异构,本文提出了一种新颖的自适应异构联邦学习方法。方法 给定一个随机生成的输入信号（例如,随机噪声）,自适应异构联邦学习直接通过对齐输出逻辑层分布来实现异构模型之间的通信,实现协作知识共享。主要优势是在不依赖额外相关数据收集或共享模型设计的情况下解决了模型异构问题。为了进一步解决数据异构问题,本文提出了在模型和样本层面上进行自适应权重更新。因此,自适应异构联邦学习（adaptive heteogeneous federated learning,AHF）允许参与者通过模型输出在无关数据上的差异和强调“有意义”的样本来学习丰富多样的知识。结果 通过在不同的联邦学习任务上使用随机噪声输入进行通信,进行了广泛的实验,显示出比竞争方法更高的域内精确度和更好的跨域泛化性能。结论 本文方法提供了一个简单而有效的基准,为异构联邦学习的未来发展奠定基础。     

线性编码联邦学习

联邦学习能够在边缘设备的协作训练中,保护边缘设备的数据隐私。而在通用联邦学习场景中,联邦学习的参与者通常由异构边缘设备构成,其中资源受限的设备会占用更长的时间,导致联邦学习系统的训练速度下降。现有方案或忽略掉队者,或根据分布式思想将计算任务进行分发,但是分发过程中涉及到原始数据的传递,无法保证数据隐私。为了缓解中小型规模的多异构设备协作训练场景下的掉队者问题,提出了编码联邦学习方案,结合线性编码的数学特性设计了高效调度算法,在确保数据隐私的同时,加速异构系统中联邦学习系统速度。同时,在实际实验平台中完成的实验结果表明,当异构设备之间性能差异较大时,编码联邦学习方案能将掉队者训练时间缩短92.85%。     

不平衡数据下预算限制的联邦学习激励机制

联邦学习的提出解决了在隐私保护下完成多客户合作的机器学习问题,而激励客户参与联邦学习是模型性能提高的一个重要前提。针对客户数据非独立同分布特征会导致联邦学习性能下降这一问题,考虑预算约束下,设计了基于单位数据成本和数据特征—EMD距离的客户端筛选方式,提出一种有效的联邦学习激励机制（EMD-FLIM）,从理论上证明了机制具有诚实性,即每个客户会诚实披露数据成本和数据分布信息,同时机制具有预算可行性,个人理性及计算有效性。实验结果显示,提出的激励机制在数据分布不平衡情况下模型精度至少能达到数据量最优选择（不考虑激励）下的 94%以上,与不考虑数据分布特征的激励机制相比较,模型精度平均可提高5%以上。     

联邦学习研究综述

联邦学习由于能够在多方数据源聚合的场景下协同训练全局最优模型,近年来迅速成为安全机器学习领域的研究热点。首先,归纳了联邦学习定义、算法原理和分类;接着,深入分析了其面临的主要威胁与挑战;然后,重点对通信效率、隐私安全、信任与激励机制3个方向的典型研究方案对比分析,指出其优缺点;最后,结合边缘计算、区块链、5G等新兴技术对联邦学习的应用前景及研究热点进行展望。     

基于区块链的联邦学习技术综述

联邦学习与区块链在应用领域、架构特点、隐私保护机制等方面具有很强的共性、互补性和契合度,近年来,一些研究与应用将两种技术结合起来,在数据隐私保护强度、数据共享激励机制、计算性能等方面取得了不少进展.为了帮助研究者掌握联邦学习结合区块链的最新研究成果与发展方向,对基于区块链的联邦学习进行了综述.首先,介绍了联邦学习技术的相关研究和存在的不足;其次,详细讨论了当前基于区块链的联邦学习的相关研究,重点从架构特点、资源分配、安全机制、激励机制等方面进行了分析;最后,总结了基于区块链的联邦学习应用在人工智能领域的未来发展趋势和需要关注的问题.     

支持隐私保护训练的高效同态神经网络

针对基于同态加密的隐私保护神经网络中存在的计算效率低和精度不足问题,提出一种三方协作下支持隐私保护训练的高效同态神经网络（HNN）。首先,为降低同态加密中密文乘密文运算产生的计算开销,结合秘密共享思想设计了一种安全快速的乘法协议,将密文乘密文运算转换为复杂度较低的明文乘密文运算;其次,为避免构建HNN时产生的密文多项式多轮迭代,并提高非线性计算精度,研究了一种安全的非线性计算方法,从而对添加随机掩码的混淆明文消息执行相应的非线性算子;最后,对所设计协议的安全性、正确性及效率进行了理论分析,并对HNN的有效性及优越性进行了实验验证。实验结果表明,相较于双服务器方案PPML,HNN的训练速度提高了18.9倍,模型精度提高了1.4个百分点。     

Multi-key privacy-preserving deep learning in cloud computing

Deep learning has attracted a lot of attention and has been applied successfully in many areas such as bioinformatics, imaging processing, game playing and computer security etc. On the other hand, deep learning usually requires a lot of training data which may not be provided by a sole owner. As the volume of data gets huge, it is common for users to store their data in a third-party cloud. Due to the confidentiality of the data, data are usually stored in encrypted form. To apply deep learning to these datasets owned by multiple data owners on cloud, we need to tackle two challenges: (i) the data are encrypted with different keys, all operations including intermediate results must be secure; and (ii) the computational cost and the communication cost of the data owner(s) should be kept minimal. In our work, we propose two schemes to solve the above problems. We first present a basic scheme based on multi-key fully homomorphic encryption (MK-FHE), then we propose an advanced scheme based on a hybrid structure by combining the double decryption mechanism and fully homomorphic encryption (FHE). We also prove that these two multi-key privacy-preserving deep learning schemes over encrypted data are secure.     

Prototyping federated learning on edge computing systems

Regarding extreme value theory, the unseen novel classes in the open-set recognition can be seen as the extreme values of training classes. Following this idea, we introduce the margin and coverage distribution to model the training classes. A novel visual-semantic embedding framework — extreme vocabulary learning (EVoL) is proposed; the EVoL embeds the visual features into semantic space in a probabilistic way. Notably, we adopt the vast open vocabulary in the semantic space to help further constraint the margin and coverage of training classes. The learned embedding can directly be used to solve supervised learning, zero-shot learning, and open set recognition simultaneously. Experiments on two benchmark datasets demonstrate the effectiveness of the proposed framework against conventional ways.     

Efficient and privacy-preserving skyline computation framework across domains

Skyline computation, which returns a set of interesting points from a potentially huge data space, has attracted considerable interest in big data era. However, the flourish of skyline computation still faces many challenges including information security and privacy-preserving concerns. In this paper, we propose a new efficient and privacy-preserving skyline computation framework across multiple domains, called EPSC. Within EPSC framework, a skyline result from multiple service providers will be securely computed to provide better services for the client. Meanwhile, minimum privacy disclosure will be elicited from one service provider to another during skyline computation. Specifically, to leverage the service provider’s privacy disclosure and achieve almost real-time skyline processing and transmission, we introduce an efficient secure vector comparison protocol (ESVC) to construct EPSC, which is exclusively based on two novel techniques: fast secure permutation protocol (FSPP) and fast secure integer comparison protocol (FSIC). Both protocols allow multiple service providers to calculate skyline result interactively in a privacy-preserving way. Detailed security analysis shows that the proposed EPSC framework can achieve multi-domain skyline computation without leaking sensitive information to each other. In addition, performance evaluations via extensive simulations also demonstrate the EPSC’s efficiency in terms of providing skyline computation and transmission while minimizing the privacy disclosure across different domains.