一种改进的双深度Q网络服务功能链部署算法

doi:10.19665/j.issn1001-2400.20230310

西安电子科技大学学报 ›› 2024, Vol. 51 ›› Issue (1): 52-59.doi: 10.19665/j.issn1001-2400.20230310

一种改进的双深度Q网络服务功能链部署算法

刘道华(), 魏丁二(), 宣贺君(), 余长鸣(), 寇丽博()

信阳师范大学计算机与信息技术学院,河南信阳 464000

收稿日期:2022-11-28 出版日期:2024-01-20 发布日期:2023-08-30
作者简介:刘道华(1974—),男,教授,博士,E-mail:ldhzzx@163.com;
魏丁二(1998—),男,信阳师范大学硕士研究生,E-mail:1186871688@qq.com;
宣贺君(1988—),男,副教授,博士,E-mail: xuanhejun0896@xynu.edu.cn;
余长鸣(1999—),女,信阳师范大学硕士研究生,E-mail:ycmxxn@163.com;
寇丽博(1998—),女,信阳师范大学硕士研究生,E-mail: 1659100219@qq.com.
基金资助:
国家自然科学基金(61572417);河南省科技攻关项目(222102210265);河南省本科高校研究性教学改革项目(2022SYJXLX061);河南省高等学校重点科研项目(22A520007);河南省研究生教育改革与质量提升工程项目(YJS2024AL104)

Improved double deep Q network algorithm for service function chain deployment

LIU Daohua(), WEI Dinger(), XUAN Hejun(), YU Changming(), KOU Libo()

School of Computer and Information Technology,Xinyang Normal University,Xinyang 464000,China

Received:2022-11-28 Online:2024-01-20 Published:2023-08-30

摘要/Abstract

摘要：

网络功能虚拟化已成为未来通信网络的关键技术,动态服务功能链的高效部署是提高网络性能迫切需要解决的问题之一。为降低通信网络服务器能量消耗以及改善通信网络服务质量,提出一种改进的双深度Q网络的动态服务功能链部署算法。由于网络状态及服务功能链的动态性,首先将服务功能链部署问题建模为马尔可夫决策过程。根据通信网络中资源的状态以及所选择的动作计算奖励函数值,对双深度Q网络进行在线训练,得到最优深度神经网络模型,从而确定最优的在线服务功能链部署策略。为解决传统深度强化学习从经验回放池中采用均匀抽取经验样本而导致神经网络学习效率低的问题,设计一种基于重要性采样的优先级经验回放方法以抽取经验样本,从而有效地避免训练样本之间的高度相关性,进一步提高离线学习神经网络的效率。仿真结果表明,所提出基于改进双深度Q网络的服务功能链部署算法能够提高奖励值,与传统的双深度Q网络算法相比,在能量消耗与阻塞率方面分别降低约19.89%~36.99%与9.52%~16.37%。

关键词: 服务功能链, 马尔科夫决策过程, 网络能耗, 双深度Q网络

Abstract:

Network Function Virtualization(NFV) has become the key technology of next generation communication.Virtual Network Function Service Chain(VNF-SC) mapping is the key issue of the NFV.To reduce the energy consumption of the communication network server and improve the quality of service,a Function Chain(SFC) deployment algorithm based on an improved Double Deep Q Network(DDQN) is proposed to reduce the energy consumption of network servers and improve the network quality of service.Due to the dynamic change of the network state,the service function chain deployment problem is modeled as a Markov Decision Process(MDP).Based on the network state and action rewards,the DDQN is trained online to obtain the optimal deployment strategy for the service function chain.To solve the problem that traditional deep reinforcement learning draws experience samples uniformly from the experience replay pool leading to low learning efficiency of the neural network,a prioritized experience replay method based on importance sampling is designed to draw experience samples so as to avoid high correlation between training samples to improve the learning efficiency of the neural network.Experimental results show that the proposed SFC deployment algorithm based on the improved DDQN can increase the reward value,and that compared with the traditional DDQN algorithm,it can reduce the energy consumption and blocking rate by 19.89%~36.99% and 9.52%~16.37%,respectively.

Key words: service function chain, Markov decision process, network energy consumption, DDQN

中图分类号:

TP393

刘道华, 魏丁二, 宣贺君, 余长鸣, 寇丽博. 一种改进的双深度Q网络服务功能链部署算法[J]. 西安电子科技大学学报, 2024, 51(1): 52-59.

LIU Daohua, WEI Dinger, XUAN Hejun, YU Changming, KOU Libo. Improved double deep Q network algorithm for service function chain deployment[J]. Journal of Xidian University, 2024, 51(1): 52-59.

图/表 8

图1

图2

图3

表1

表2

图4

图5

图6

参考文献 15

[1]	乔文欣, 卢昱, 刘益岑, 等. 空天地协同的边缘云服务功能链动态编排方法[J]. 西安电子科技大学学报, 2022, 49(2):79-88.
	QIAO Wenxin, LU Yu, LIU Yicen, et al. Dynamic Scheduling Method for Service Function Chains in Space Air Terrestrial Aided Edge Cloud Network[J]. Journal of Xidian University, 2022, 49(2):79-88.
[2]	SHAH H A, ZHAO L. Multiagent Deep-Reinforcement-Learning-Based Virtual Resource Allocation Through Network Function Virtualization in Internet of Things[J]. IEEE Internet of Things Journal, 2021, 8(5):3410-3421. doi: 10.1109/JIoT.6488907
[3]	YANG S, LI F, TRAJANOVSI S, et al. Recent Advances of Resource Allocation in Network Function Virtualization[J]. IEEE Transactions on Parallel and Distributed Systems, 2021, 32(2):259-314.
[4]	FAN W, CUI Q, LI X, et al. On Credibility-Based Service Function Chain Deployment[J]. IEEE Open Journal of the Computer Society, 2021, 2:152-163. doi: 10.1109/OJCS.2021.3064887
[5]	CHEN J, CHEN X, CHEN J, et al. A Lightweight SFC Embedding Framework in SDN/NFV-Enabled Wireless Network Based on Reinforcement Learning[J]. IEEE Systems Journal, 2022, 16(3):3817-3828. doi: 10.1109/JSYST.2021.3111972
[6]	FARKIANI S, BAKHSHI B, MIRHASSANI S A, et al. Prioritized Deployment of Dynamic Service Function Chains[J]. IEEE/ACM Transactions on Networking, 2021, 29(3):979-993. doi: 10.1109/TNET.2021.3055074
[7]	KARIMZADEH F M, MANSOURI S V, NIYATO D. A Dynamic Reliability-Aware Service Placement for Network Function Virtualization(NFV)[J]. IEEE Journal on Selected Areas in Communications, 2020, 38(2):318-333. doi: 10.1109/JSAC.49
[8]	KHEBBACHE S, HADJI M, ZEGHLACHE D. Scalable and Cost-Efficient Algorithms for VNF Chaining and Placement Problem[C]//Proceedings of the 20th Conference on Innovations in Clouds,Internet and Networks(ICIN).Piscataway:IEEE, 2017:92-99.
[9]	PEI J, HONG P, XUE K, et al. Resource Aware Routing for Service Function Chains in SDN and NFV-Enabled Network[J]. IEEE Transactions on Services Computing, 2021, 14(4):985-997. doi: 10.1109/TSC.2018.2849712
[10]	VIZARRETA P, CONDOLUCI M, MACHUCA C M, et al. QoS-Driven Function Placement Reducing Expenditures in NFV Deployments[C]//Proceedings of the 2017 IEEE International Conference on Communications(ICC). Piscataway:IEEE, 2017:1-7.
[11]	魏亮, 黄韬, 张娇, 等. 基于强化学习的服务链映射算法[J]. 通信学报, 2018, 39(1):90-100. doi: 10.11959/j.issn.1000-436x.2018002
	WEI Liang, HUANG Tao, ZHANG Jiao, et al. Service Chain Mapping Algorithm Based on Reinforcement Learning[J]. Journal on Communications, 2018, 39(1):90-100. doi: 10.11959/j.issn.1000-436x.2018002
[12]	JIN Q, GE S, ZENG J, et al. ScaRL:Service Function Chain Allocation Based on Reinforcement Learning in Mobile Edge Computing[C]//Proceedings of the 7th International Conference on Advanced Cloud and Big Data(CBD). Piscataway:IEEE, 2019:327-332.
[13]	陈卓, 冯钢, 何颖, 等. 运营商网络中基于深度强化学习的服务功能链迁移机制[J]. 电子与信息学报, 2020, 42(9):2173-2179.
	CHEN Zhuo, FENG Gang, HE Ying, et al. Service Function Chain Migration Mechanism Based on Deep Reinforcement Learning in Operator networks[J]. Journal of Electronics & Information Technology, 2020, 42(9):2173-2179.
[14]	HOU Y, LIU L, WEI Q, et al. A Novel DDPG Method with Prioritized Experience Replay[C]//IEEE International Conference on Systems,Man,and Cybernetics(SMC). Piscataway:IEEE, 2017,316-321.
[15]	金明, 李琳琳, 张文瑾, 等. 基于深度强化学习的服务功能链映射算法[J]. 计算机应用研究, 2020, 37(11):3456-3460.
	JIN Ming, LI Linlin, ZHANG Wenjin, et al. Mapping Algorithm of Service Function Chain Based on Deep Reinforcement Learning[J]. Computer Application Research, 2020, 37(11):3456-3460.

仿真参数	数值	仿真参数	数值
服务器数量	5	κ	0.01
折扣因子γ	0.9	β	0.4
调节优先程度α	0.6	η_v	0.6
学习率ρ	0.001	ε	0.8
批量大小	64	时延限制/ms	[50,90]

网络层	卷积核参数	步长参数	卷积核个数	激活函数
卷积层1	3×3	1	64	ReLU
卷积层2	3×3	1	128	ReLU
全连接层1			256	ReLU
全连接层2			5	Linear

一种改进的双深度Q网络服务功能链部署算法

Improved double deep Q network algorithm for service function chain deployment

RichHTML

PDF (PC)

赞

可视化

摘要/Abstract

引用本文

使用本文

图/表 8

参考文献 15

相关文章 1

Metrics

本文评价

推荐阅读 0