可实现双向自适应差分隐私的联邦学习方案

doi:10.19665/j.issn1001-2400.20230706

摘要/Abstract

摘要：

随着个人数据的爆发式增长,基于差分隐私的联邦学习模型可用于解决数据孤岛问题和保护用户数据隐私,参与者通过训练本地数据,将添加噪声后的参数共享到中心服务器进行聚合,实现分布式机器学习训练。此过程中存在两方面问题:① 中心服务器广播参数的过程中数据信息仍未受到保护,有泄露用户隐私的风险;② 对参数过度添加噪声会导致参数聚合质量降低,影响最终联邦学习的模型精度。为解决以上问题,提出了一种可实现双向自适应差分隐私的联邦学习方案(FedBADP),对客户端和中心服务器之间传输的梯度进行自适应加噪,在保护数据安全的同时不影响模型准确率。考虑到参与者硬件设备的性能限制,文中对其梯度进行采样以减少通信开销,并在客户端和中心服务器使用均方根传递加速模型的收敛提高模型精度。实验结果证明,文中提出的模型框架在保持较好准确率的同时,也增强了用户的隐私保护能力。

关键词: 双向自适应噪声, 均方根传递, 采样, 差分隐私, 联邦学习

Abstract:

With the explosive growth of personal data,the federated learning based on differential privacy can be used to solve the problem of data islands and preserve user data privacy.Participants share the parameters with noise to the central server for aggregation by training local data,and realize distributed machine learning training.However,there are two defects in this model:on the one hand,the data information in the process of parameters broadcasting by the central server is still compromised,with the risk of user privacy leakage;on the other hand,adding too much noise to parameters will reduce the quality of parameter aggregation and affect the model accuracy of federated learning.In order to solve the above problems,a bidirectional adaptive differential privacy federated learning scheme(Federated Learning Approach with Bidirectional Adaptive Differential Privacy,FedBADP) is proposed,which can adaptively add noise to the gradients transmitted by participants and central servers,and keep data security without affecting the model accuracy.Meanwhile,considering the performance limitations of the participants hardware devices,this model samples their gradients to reduce the communication overhead,and uses the RMSprop to accelerate the convergence of the model on the participants and central server to improve the accuracy of the model.Experiments show that our novel model can enhance the user privacy preserving while maintaining a good accuracy.

Key words: bidirectional adaptive noise, RMSprop, sampling, differential privacy, federated learning

中图分类号:

TP309

李洋, 徐进, 朱建明, 王友卫. 可实现双向自适应差分隐私的联邦学习方案[J]. 西安电子科技大学学报, 2024, 51(3): 158-169.

LI Yang, XU Jin, ZHU Jianming, WANG Youwei. Bidirectional adaptive differential privacy federated learning scheme[J]. Journal of Xidian University, 2024, 51(3): 158-169.

图/表 7

表1

图1

表2

图2

图3

图4

图5

参考文献 25

[1]	MCMAHAN H B, MOORE E, RAMAGE D, et al. Communication-Efficient Learning of Deep Networks from Decentralized Data[C]//Proceedings of the 20th International Conference on Artificial Intelligence and Statistics. New York: PMLR, 2017:1273-1282.
[2]	JIANG J, KANTARCI B, OKTUG S F, et al. Federated Learning in Smart City Sensing:Challenges and Opportunities[J]. Sensors, 2020, 20(21):6230.
[3]	XU J, GLICKSBERG B S, SU C, et al. Federated Learning for Healthcare Informatics[J]. Journal of Healthcare Informatics Research 5, 2021, 5(1):1-19.
[4]	TU X, ZHU K, LUONG N C, et al. Incentive Mechanisms for Federated Learning:From Economic and Game Theoretic Perspective(2021)[J/OL].[2021-11-20]. https://arxiv.org/abs/2111.11850v1.
[5]	陈嘉翊, 孙晨雨, 周欣桐, 等. 基于联邦学习和同态加密的电力数据预测模型本地保护[J]. 信息安全研究, 2023, 9(3):228-234.
	CHEN Jiayi, SUN Chenyu, ZHOU Xintong, et al. Local Privacy Protection for Power Data Prediction Model Based on Federated Learning and Homomorphic Encryption[J]. Information Security Research, 2023, 9(3):228-234.
[6]	徐花, 田有亮. 差分隐私下的权重社交网络隐私保护[J]. 西安电子科技大学学报, 2022, 49(1):17-25.
	XU Hua, TIAN Youliang. Protection of Privacy of the Weighted Social Network under Differential Privacy[J]. Journal of Xidian University, 2022, 49(1):17-25.
[7]	WANG F, XIE M, TAN Z, et al. Preserving Differential Privacy in Deep Learning Based on Feature Relevance Region Segmentation[J]. IEEE Transactions on Emerging Topics in Computing, 2023, 12(1):307-315.
[8]	FU J, CHEN Z, HAN X. Adap DP-FL:Differentially Private Federated Learning with Adaptive Noise[C]//2022 IEEE International Conference on Trust,Security and Privacy in Computing and Communications(TrustCom).Piscataway:IEEE, 2022:656-663.
[9]	粟勇, 刘文龙, 刘圣龙, 等. 基于安全洗牌和差分隐私的联邦学习模型安全防护方法[J]. 信息安全研究, 2022, 8(3):270-276.
	SU Yong, LIU Wenlong, LIU Shenglong, et al. Secure Protection Method for Federated Learning Model Based on Secure Shuffling and Differential Privacy[J]. Information Security Research, 2022, 8(3):270-276.
[10]	晏燕, 董卓越, 徐飞, 等. 一种Hilbert编码的本地化位置隐私保护方法[J]. 西安电子科技大学学报, 2022, 50(2):147-160.
	YAN Yan, DONG Zhuofei, XU Fei, et al. Localized Location Privacy Protection Method Using the Hilbert Encoding[J]. Journal of Xidian University, 2022, 50(2):147-160.
[11]	KONEN J, MCMAHAN H B, YU F X, et al. Federated Learning:Strategies for Improving Communication Efficiency(2016)[J/OL].[2016-10-18].https://arxiv.org/abs/1610.05492.
[12]	LI T, SAHU A K, ZAHEER M, et al. Federated Optimization in Heterogeneous Networks(2018)[J/OL].[2018-12-14]. https://arxiv.org/abs/1812.06127v5.
[13]	LIU W, CHEN L, CHEN Y, et al. Accelerating Federated Learning via Momentum Gradient Descent[J]. IEEE Transactions on Parallel and Distributed Systems, 2020, 31(8):1754-66.
[14]	LIU X, LI Y, WANG Q, et al. Sparse Personalized Federated Learning via Maximizing Correlation(2021)[J/OL].[2021-07-12]. https://arxiv.org/abs/2107.05330v3.
[15]	MELIS L, SONG C, CRISTOFARO E D, et al. Inference Attacks Against Collaborative Learning(2018)[J/OL].[2018-05-10]. https://arxiv.org/abs/1805.04049v1.
[16]	ABADI M, CHU A, GOODFELLOW I, et al. Deep Learning with Differential Privacy[C]//Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security. New York: ACM, 2016:308-318.
[17]	WEI K, LI J, DING M, et al. Federated Learning with Differential Privacy:Algorithms and Performance Analysis[J]. IEEE Transactions on Information Forensics and Security, 2020, 15:3454-3469.
[18]	WEI K, LI J, DING M, et al. User-Level Privacy-Preserving Federated Learning:Analysis and Performance Optimization[J]. IEEE Transactions on Mobile Computing, 2022, 21(9):3388-3401.
[19]	WU X, ZHANG Y, SHI M, et al. AnAdaptive Federated Learning Scheme with Differential Privacy Preserving[J]. Future Generation Computer Systems, 2022, 127:362-72.
[20]	XU Z, SHI S, LIU A X, et al. An Adaptive and Fast Convergent Approach to Differentially Private Deep Learning[C]//Proceedings of the IEEE INFOCOM 2020:International Conference on Computer Communications(INFOCOM). Piscataway:IEEE, 2020:1867-1876.
[21]	KUMAR G, PRIYA G, DILEEP M, et al. Image Deconvolution using Deep Learning-based Adam Optimizer[C]//2022 6th International Conference on Electronics,Communication and Aerospace Technology.Piscataway:IEEE, 2022:901-904.
[22]	XIANG L, YANG J, LI B. Differentially-Private Deep Learning from an Optimization Perspective[C]//Proceedings of the IEEE Conference on Computer Communications. Piscataway:IEEE, 2019:559-567.
[23]	DWORK C. Differential Privacy[C].Proceedings of the 33rd International Conference on Automata,Languages and Programming. Heidelberg:Springer, 2006:1-10.
[24]	LI N, LYU M, SU D, et al. Differential Privacy:From Theory to Practice[J]. Synthesis Lectures on Information Security,Privacy,and Trust, 2016, 8(4):1-138.
[25]	GEYER R C, KLEIN T, NABI M. Differentially Private Federated Learning:A Client Level Perspective(2017)[J/OL].[2017-12-20]. https://arxiv.org/abs/1712.07557v2.

文献	加噪方式
FedBADP	自适应双向加噪
ABADI等^[16]	客户端单向加噪
WEI K等^[17]	双向加噪
KUMAR G等^[21]	中心服务器加噪
XIANG L等^[22]	自适应加噪

算法	通信轮数	客户端采样率	样本采样率	准确率/%	差分隐私
DP-SGD^[16] FedBADP1	700 81	1.00 0.22	0.01 0.10	97.00 96.85	(8,1e-5)-DP (8,1e-288)-DP
CRD^[17] FedBADP2	30 32	0.60 0.50	1.00 0.10	90.00 90.91	(8,1e-5)-DP (8,1e-288)-DP
DPOPT^[22] FedBADP3	100 60	1.00 1.00	1.00 1.00	96.00 96.29	(5.48,1e-4)-DP (5.48,1e-135)-DP
CDP-FL^[25] FedBADP4	54 42	0.22 0.22	1.00 0.10	92.00 94.40	(8,1e-5)-DP (8,1e-288)-DP