面向多网联无人机的MADRL协同路径规划算法

doi:10.19665/j.issn1001-2400.20250102

摘要/Abstract

摘要：

针对多架网联无人机执行多个动态目标点巡航监测任务的协同路径规划问题,文中提出了一种基于多智能体深度强化学习的协同动态目标分配与路径规划算法。具体而言,首先对多架CUAV对地通信系统环境建模,在CUAV与蜂窝网络通信连接约束、防碰撞等移动性约束下,构建多架CUAV任务完成总时间和总中断时间加权和最小化优化问题。然后把优化问题离散化并转为马尔可夫博弈。为解决所构建的组合优化问题,提出了基于深度确定性策略梯度的系列算法,有效地处理CUAV之间的合作与竞争关系,且引入一个共享经验池,以提高算法的稳定性和学习效率。仿真实验验证了所提出的MADRL框架在多架CUAV协同路径规划的可行性、有效性和鲁棒性。研究表明:多智能体双延迟-确定策略梯度算法在动态目标位置的协同目标分配与路径规划中具有更好的效果。

关键词: 无人机, 强化学习, 网联无人机, 多智能体深度强化学习, 路径规划, 动态目标分配

Abstract:

In this paper,a collaborative dynamic target assignment and path planning algorithm based on Multi-Agent Deep Reinforcement Learning (MADRL) is proposed to address the cooperative path planning of multi-CUAV for multi-dynamic target point monitoring.Specifically,the communication system environment for multi-CUAV-to-ground is modeled,and the weighted sum of the total completion time and total interruption time of multi-CUAV minimization optimization is formulated which is subject to the communication connectivity constraints with the cellular network,the collision avoidance constraints,etc.Then the formulated optimization problem is discretized and transformed into the Markov game.To tackle the formulated combinatorial optimization problem,Deep Deterministic Policy Gradient (DDPG) series algorithms are proposed to effectively handle the cooperative and competitive behaviors among multi-CUAV.Meanwhile,a shared experience pool is introduced to improve the stability and learning efficiency of the proposed algorithm.The feasibility,effectiveness,and robustness of the proposed MADRL framework are demonstrated by numerical simulations,which indicates that the multi-agent twin delayed deep deterministic policy gradient algorithm (MATD3) exhibits a superior performance for cooperative target assignment and path planning with dynamic target locations.

Key words: unmanned aerial vehicle, reinforcement learning, cellular-connected unmanned aerial vehicle, multi-agent deep reinforcement learning, path planning, dynamic target assignment

中图分类号:

TN92

李安, 余传鑫, 陈成. 面向多网联无人机的MADRL协同路径规划算法[J]. 西安电子科技大学学报, 2025, 52(3): 163-175.

LI An, YU Chuanxin, CHEN Cheng. Multi-agent deep reinforcement learning assisted cooperative path planning for the multi-cellular-connected unmanned aerial vehicle[J]. Journal of Xidian University, 2025, 52(3): 163-175.

图/表 9

图1

图2

图3

图4

图5

图6

图7

图8

图9

参考文献 32

[1]	ZHANG C, XIAO Z, ZHANG L, et al.Collaborative Multi-Agent Jamming Deceiving for UAV-Assisted Wireless Communications[C]// 2024 International Wireless Communications and Mobile Computing (IWCMC). Piscataway:IEEE,2024:549-554.
[2]	GUO K, WU M, LI X, et al. Joint Trajectory and Beamforming Optimization for Federated DRL-Aided Space-Aerial-Terrestrial Relay Networks with RIS and RSMA[J]. IEEE Transactions on Wireless Communications, 2024:23(12):18456-18471.
[3]	柴蓉, 崔相霖, 孙瑞锦, 等. 基于通感融合的无人机预编码及飞行轨迹设计[J]. 电子与信息学报, 2024, 46(4):1266-1275.
	CHAI Rong, CUI Xianglin, SUN Ruijin, et al. Precoding and Trajectory Design of Unmanned Aerial Vehicle Based on Joint Communication and Sensing[J]. Journal of Electronics & Information Technology, 2024, 46(4):1266-1275.
[4]	李鹏展. 针对5G网联无人机的区域基站选址优化[D]. 北京: 清华大学, 2024.
[5]	ZENG S, XU X, ZENG Y, et al. CKM-Assisted LoS Identification and Predictive Beamforming for Cellular-Connected UAV[C]// ICC 2023-IEEE International Conference on Communications. Piscataway:IEEE,2023:2877-2882.
[6]	CHEN Y J, HUANG D Y. Trajectory Optimization for Cellular-enabled UAV with Connectivity Outage Constraint[J]. IEEE Access, 2020,8:29205-29218.
[7]	YANG H, ZHANG J, SONG S H, et al. Connectivity-aware UAV Path Planning with Aerial Coverage Maps[C]// 2019 IEEE Wireless Communications and Networking Conference (WCNC). Piscataway:IEEE,2019:1-6.
[8]	KHAMIDEHI B, SOUSA E S. Federated Learning for Cellular-connected UAVs:Radio Mapping and Path Planning[C]// 2020 IEEE Global Communications Conference. Piscataway:IEEE,2020:1-6.
[9]	ZHANG S, ZENG Y, ZHANG R. Cellular-enabled UAV Communication:A Connectivity-Constrained Trajectory Optimization Perspective[J]. IEEE Transactions on Communications, 2018, 67(3):2580-2604.
[10]	GALKIN B, AMER R, FONSECA E, et al. Intelligent Base Station Association for UAV Cellular Users:A Supervised Learning Approach[C]// 2020 IEEE 3rd 5G World Forum (5GWF). Piscataway:IEEE,2020:383-388.
[11]	MEI W, ZHANG R. Cooperative Downlink Interference Transmission and Cancellation for Cellular-connected UAV:A Divide-and-Conquer Approach[J]. IEEE Transactions on Communications, 2019, 68(2):1297-1311.
[12]	RAJA G, SIVAGANESH B, RAVICHANDRAN V, et al. AI-Empowered UAV Trajectory Optimization in 6G Aerial Networks[C]// 2023 IEEE Global Communications Conference. Piscataway:IEEE,2023:7285-7290.
[13]	CUI H, ZHANG N, LIU P. Trajectory Optimization for 6G-UAV Based on Deep Reinforcement Learning[J]. IEEE Transactions on Vehicular Technology, 2024, 73(11):17935-17939.
[14]	ZHAO H, HAO Q, HUANG H, et al. Online Trajectory Optimization for Energy-Efficient Cellular-Connected UAVs with Map Reconstruction[J]. IEEE Transactions on Vehicular Technology, 2024, 73(3):3445-3456.
[15]	ZHAN C, HU H, LIU Z, et al. Aerial Video Streaming over 3D Cellular Networks:An Environment and Channel Knowledge Map Approach[J]. IEEE Transactions on Wireless Communications, 2024, 23(2):1432-1446.
[16]	FONTANESI G, ZHU A, ARVANEH M, et al. A Transfer Learning Approach for UAV Path Design with Connectivity Outage Constraint[J]. IEEE Internet of Things Journal, 2023, 10(6):4998-5012.
[17]	HAO Q, ZHAO H, HUANG H, et al. Deep Reinforcement Learning Aided Online Trajectory Optimization of Cellular-Connected UAVs with Offline Map Reconstruction[C]// 2023 IEEE 97th Vehicular Technology Conference (VTC2023-Spring). Piscataway:IEEE,2023:1-5.
[18]	RIEDMILLER M, HAFNER R, LAMPE T, et al. Learning by Playing Solving Sparse Reward Tasks from Scratch[C]// International Conference on Machine Learning. New York: PMLR,2018:4344-4353.
[19]	WANG X, GURSOY M C, ERPEK T, et al. Jamming-Resilient Path Planning for Multiple UAVs via Deep Reinforcement Learning[C]// 2021 IEEE International Conference on Communications Workshops (ICC Workshops). Piscataway:IEEE,2021:1-6.
[20]	CHEN Y J, HUANG D Y. Joint Trajectory Design and BS Association for Cellular-connected UAV:An Imitation-Augmented Deep Reinforcement Learning Approach[J]. IEEE Internet of Things Journal, 2021, 9(4):2843-2858.
[21]	LIU B, NI W, LIU R P, et al. Decentralized,Privacy-Preserving Routing of Cellular-Connected Unmanned Aerial Vehicles for Joint Goods Delivery and Sensing[J]. IEEE Transactions on Intelligent Transportation Systems, 2023, 24(9):9627-9641.
[22]	ZHAN C, HU H, LIU Z, et al. Interference-Aware Online Optimization for Cellular-Connected Multiple UAV Networks with Energy Constraints[J]. IEEE Transactions on Mobile Computing, 2024, 23(12):13804-13820.
[23]	WU Q, ZENG Y, ZHANG R. Joint Trajectory and Communication Design for Multi-UAV Enabled Wireless Networks[J]. IEEE Transactions on Wireless Communications, 2018, 17(3):2109-2121.
[24]	PAN Z, ZHANG C, XIA Y, et al. An Improved Artificial Potential Field Method for Path Planning and Formation Control of the Multi-UAV Systems[J]. IEEE Transactions on Circuits and Systems II:Express Briefs, 2022, 69(3):1129-1133.
[25]	LOWE R, WU Y, Tamar A, et al. Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments[C]// The 31st International Conference on Neural Information Processing Systems (NIPS'17). New York: Curran Associates Inc.,2017:6382-6393.
[26]	GUPTA A, PACCHIANO A, ZHAI Y, et al. Unpacking Reward Shaping:Understanding the Benefits of Reward Engineering on Sample Complexity[J]. Advances in Neural Information Processing Systems, 2022,35:15281-15295.
[27]	SUTTON R S, BARTO A G. Reinforcement Learning:An introduction[M]. Cambridge,MA: MIT press, 2018.
[28]	ZHANG Q, LIANG Y C, POOR H V. Intelligent User Association for Symbiotic Radio Networks Using Deep Reinforcement Learning[J]. IEEE Transactions on Wireless Communications, 2020, 19(7):4535-4548.
[29]	International Telecommunication Union (ITU-R). Propagation Data and Prediction Methods Required for the Design of Terrestrial Broadband Radio Access Systems Operating in a Frequency Range From 3 to 60 GHz:Rec.1410-5[S]. Geneva: International Telecommunication Union,2013:1410-1415.
[30]	3GPP. Study on 3D Channel Model for LTE:TR 36.873 V12.7.0[S]. Sophia Antipolis:3GPP, 2017.
[31]	3GPP. Technical Specification Group Radio Access Network:Study on Enhanced LTE Support for Aerial Vehicles:TR 36.777 V15.0.0[S]. Sophia Antipolis:3GPP, 2017.
[32]	ZENG Y, XU X, JIN S, et al. Simultaneous Navigation and Radio Mapping for Cellular-Connected UAV with Deep Reinforcement Learning[J]. IEEE Transactions on Wireless Communications, 2021, 20(7):4205-4220.