采用DDPG的联合波束成形和功率控制算法

doi:10.19665/j.issn1001-2400.2022.04.006

西安电子科技大学学报 ›› 2022, Vol. 49 ›› Issue (4): 39-48.doi: 10.19665/j.issn1001-2400.2022.04.006

采用DDPG的联合波束成形和功率控制算法

李中捷^1,²(),高伟^1,²(),熊吉源^1,²(),李江虹^1,²()

1.中南民族大学电子信息工程学院,湖北武汉 430074
2.中南民族大学智能无线通信湖北重点实验室,湖北武汉 430074

收稿日期:2021-04-16 出版日期:2022-08-20 发布日期:2022-08-15
通讯作者: 高伟,熊吉源,李江虹
作者简介:李中捷(1974—),男,教授,博士,E-mail: lizhongjie@mail.scuec.edu.cn
基金资助:
国家自然科学基金(61379028);国家自然科学基金(61671483);湖北省自然科学基金(2016CFA089);中央高校基本科研业务费专项资金(CZY19003)

DDPG method for joint beamforming and power control in mmwave communication

LI Zhongjie^1,²(),GAO Wei^1,²(),XIONG Jiyuan^1,²(),LI Jianghong^1,²()

1. College of Electronic and Information Engineering,South-Central University for Nationalities,Wuhan 430074,China
2. Hubei Key Laboratory of Intelligent Wireless Communications,South-Central University for Nationalities,Wuhan 430074,China

Received:2021-04-16 Online:2022-08-20 Published:2022-08-15
Contact: Wei GAO,Jiyuan XIONG,Jianghong LI

摘要/Abstract

摘要：

针对现有波束成形算法性能,大程度依赖信道状态信息质量而不适用于快速变化的实际系统和忽略功率控制问题,导致用户间干扰严重,从而降低通信链路频谱效率的问题,提出了基于深度强化学习的联合波束成形和功率控制算法,在无须完整信道状态信息的情况下对波束成形矩阵和功率控制问题进行联合求解。同时提出信息交互协议利于基站了解环境信息,并设计集中式训练分布式执行结构的双模型系统对联合优化问题进行求解。首先基站收集本地样本并上传至云端,云端接受基站上传的本地样本后,采用深度Q学习对波束成形进行设计;然后用深度确定性策略梯度算法取代深度Q学习求解功率控制问题,从而解决深度Q学习不适用于连续变量的问题。云端模型训练完成后,广播给所有基站进行分布式执行获取本地样本。仿真结果表明,在多用户通信环境中该算法的频谱效率优于传统波束成形算法和基于深度Q学习的联合波束成形和功率控制算法,从而验证了该算法的有效性。

关键词: 深度强化学习, 深度确定性策略梯度, 波束成形, 功率控制

Abstract:

The majority of existing beamforming algorithms rely heavily on the quality of instantaneous channel state information (CSI),which is unsuitable for practical systems and ignores power control issues,resulting in serious inter-user interference and lowering the spectrum efficiency.A deep reinforcement learning-based joint beamforming and power control technique is proposed to jointly tackle the beamforming design and power control problems without the requirement for perfect CSI.First,an information exchange protocol is proposed to facilitate the base station to understand environmental information,and a dual-model system with a centralized training distributed execution structure is designed to solve the joint optimization problem.The cloud utilizes Deep Q-Learning (DQN) to design the beamforming after receiving the local samples uploaded by the base station,which is collected and uploaded to the cloud by the base station.Considering that deep Q learning is not applicable to continuous variables,we employ the deep deterministic strategy gradient algorithm(DDPG) to solve the power control problem.The cloud model is broadcast to all base stations for distributed execution in order to acquire local samples once the training is completed.Simulation results show that the proposed scheme for spectral efficiency optimization significantly outperforms the traditional beamforming algorithm.

Key words: deep reinforcement learning, deep deterministic policy gradient, beamforming, power control

中图分类号:

TN928

李中捷,高伟,熊吉源,李江虹. 采用DDPG的联合波束成形和功率控制算法[J]. 西安电子科技大学学报, 2022, 49(4): 39-48.

LI Zhongjie,GAO Wei,XIONG Jiyuan,LI Jianghong. DDPG method for joint beamforming and power control in mmwave communication[J]. Journal of Xidian University, 2022, 49(4): 39-48.

图/表 9

图1

图2

图3

图4

表1

表2

图5

图6

图7

参考文献 19

[1]	ZHANG N, WANG J, KANG G, et al. Uplink Nonorthogonal Multiple Access in 5G Systems[J]. I IEEE Communications Letters, 2016, 20 (3):458-461.
[2]	DING Z, FAN P, POOR H V. Random Beamforming in Millimeterwave NOMA Networks[J]. IEEE Access, 2017, 5:7667-7681. doi: 10.1109/ACCESS.2017.2673248
[3]	HAN S, CHIH-LIN I, XIE T, et al. Achieving High Spectrum Efficiency on High Speed Train for 5G New Radio and Beyond[J]. IEEE Wireless Communications, 2019, 26(5):62-69.
[4]	谢垚, 黄默, 王长元, 等. 一种改进的宽波束线阵雷达物位测量算法[J]. 西安电子科技大学学报, 2021, 48(2):165-172.
	XIE Yao, HUANG Mo, WANG Changyuan, et al. Improved Algorithm for Wide Beam Linear Array Radar Level Meter Measurement[J]. Journal of Xidian University, 2021, 48(2):165-172.
[5]	SHEN K, YU W. Fractional Programming for Communication Systems-Part I:Power Control and Beamforming[J]. IEEE Transactions on Signal Processing, 2018, 66(10):2616-2630. doi: 10.1109/TSP.2018.2812733
[6]	SHI Q, RAZAVIYAYN M, LUO Z Q, et al. An Iteratively Weighted MMSE Approach to Distributed Sum-utility Maximization for a MIMO Interfering Broadcast Channel[J]. IEEE Transactions on Signal Processing, 2011, 59(9):4331-4340. doi: 10.1109/TSP.2011.2147784
[7]	NASIRY S, GUO D. Multi-Agent Deep Reinforcement Learning for Dynamic Power Allocation in Wireless Networks[J]. IEEE Journal on Selected Areas in Communications, 2019, 37(10):2239-2250. doi: 10.1109/JSAC.2019.2933973
[8]	YE H, LI G Y, JUANG B H F. Deep Reinforcement Learning Based Resource Allocation for V2V Communications[J]. IEEE Transactions on Vehicular Technology, 2019, 68(4):3163-3173. doi: 10.1109/TVT.2019.2897134
[9]	LEE C, KIM Y H. Receive Beamforming and Resource Allocation for Wireless Powered Non-Orthogonal Multiple Access[J]. IEEE Transactions on Vehicular Technology, 2020, 69(4):4563-4568. doi: 10.1109/TVT.2020.2976887
[10]	XIAO Z, ZHU L, CHOI J, et al. Joint Power Allocation and Beamforming for Non-Orthogonal Multiple Access (NOMA) in 5G Millimeter Wave Communications[J]. IEEE Transactions on Wireless Communications, 2018, 17(5):2961-2974. doi: 10.1109/TWC.2018.2804953
[11]	XU J, AI B, CHEN L. Joint Beamforming and Power Allocation in Millimeter-Wave High-Speed Railway Systems[C]// Proceedings of the 2020 IEEE Global Communications Conference.Piscataway:IEEE, 2020:1-6.
[12]	LILLICRAPT P, HUNT J J, PRITZEL A, et al. Continuous Control with Deep Reinforcement Learning[J]. Computer Science, 2015, 8(6):A187.
[13]	MISMANF B, EVANS B L, ALKHATEEB A. Deep Reinforcement Learning for 5G Networks:Joint Beamforming,Power Control,and Interference Coordination[J]. IEEE Transactions on Communications, 2020, 66(3):1581-1592.
[14]	GE J, LIANG Y C, JOUNG J, et al. Deep Reinforcement Learning for Distributed Dynamic MISO Downlink-Beamforming Coordination[J]. IEEE Transactions on Communications, 2020, 68(10):6070-6085. doi: 10.1109/TCOMM.2020.3004524
[15]	DONG M, TONG L, SADLER B M. Optimal Insertion of Pilot Symbols for Transmissions over Time-Varying Flat Fading Channels[J]. IEEE Transactions on Signal Processing, 2004, 52(5):1403-1418. doi: 10.1109/TSP.2004.826182
[16]	ARULKUMARAN K, DEISENROTH M P, BRUNDAGE M, et al. Deep Reinforcement Learning:A Brief Survey[J]. IEEE Signal Processing Magazine, 2017, 34(6):26-38. doi: 10.1109/MSP.2017.2743240
[17]	ZOU W, CUI Z, LI B, et al. Beamforming Codebook Design and Performance Evaluation for 60GHz Wireless Communication[C]// 2011 11th International Symposium on Communications & Information Technologies.Piscataway:ISCIT, 2011:30-35.
[18]	ZHUANG B, GUO D, HONIG M L. Energy-efficient Cell Activation,User Association,and Spectrum Allocation in Heterogeneous Networks[J]. IEEE Journal on Selected Areas in Communications, 2016, 34(4):823-831. doi: 10.1109/JSAC.2016.2544478
[19]	CYBENKO G. Approximation by Superpositions of a Sigmoidal Function[J]. Mathematics of Control,Signals and Systems, 1989, 2(4):303-314. doi: 10.1007/BF02551274

参数	参数值
蜂窝网络半径/m	500
(基站)/(用户)天线数量	32/1
最大发射功率/dBm	38
对数正态分布的标准差/dB	10
噪声功率/dBm	-114
拓展角Δ/(°)	3
阈值η	5
相关系数ρ	0.64

参数	参数值
探索率(ε)	ε₍₀₎=0.2,ε_min=10^-2
学习率	10^-3
折扣因子	γ=0.5
小批量数据M_b	32
经验池数	500
周边基站个数	5
隐藏层单元个数	200,100,50
权重赋值间隔T_u	50

采用DDPG的联合波束成形和功率控制算法

DDPG method for joint beamforming and power control in mmwave communication

RichHTML

PDF (PC)

赞

可视化

摘要/Abstract

引用本文

使用本文

图/表 9

参考文献 19

相关文章 15

Metrics

本文评价

推荐阅读 0

[1]	宁晓晗,雷维嘉. 能量收集多天线发送机的功率控制和天线选择[J]. 西安电子科技大学学报, 2022, 49(3): 83-92.
[2]	赫彬,苏洪涛. 一种多目标与多基地雷达之间的博弈策略[J]. 西安电子科技大学学报, 2021, 48(2): 125-132.
[3]	施焘,韩雪峰,陈文. 一种高铁场景下的接收机设计方案[J]. 西安电子科技大学学报, 2020, 47(6): 30-36.
[4]	仲伟志, 徐磊, 朱秋明, 陈小敏, 周建江. 面向天线互耦效应抑制的波束预编码算法[J]. 西安电子科技大学学报, 2018, 45(6): 80-85.
[5]	唐俊林;曾媛;岳光荣;李少谦. 毫米波相阵多径信道中的迭代波束成形方法[J]. 西安电子科技大学学报, 2018, 45(4): 143-148.
[6]	唐俊林;曾媛;岳光荣;李少谦. 毫米波相控阵的一种迭代波束成形方法[J]. 西安电子科技大学学报, 2017, 44(6): 122-128.
[7]	周彦果;张海林;陈瑞瑞. 毫微微小区网络中采用双层博弈的资源分配[J]. 西安电子科技大学学报, 2017, 44(4): 62-68.
[8]	乔俊峰;刘三阳;齐小刚. 无线传感器网络中基于鲁棒优化的功率控制[J]. 西安电子科技大学学报, 2016, 43(5): 81-87.
[9]	夏玉杰;任光亮. 波束成形辅助的连续干扰消除OFDMA初始测距算法[J]. J4, 2013, 40(4): 79-84+154.
[10]	任光亮;任晓娜. MIMO-OFDMA系统中的一种新的初始测距方法[J]. J4, 2013, 40(4): 1-5+41.
[11]	袁韵洁李波闫中江. 基于功率控制和动态载波侦听的信道预约多址接入协议[J]. J4, 2013, 40(2): 181-186.
[12]	袁韵洁;李波;闫中江. 一种自适应功率控制的信道预约多址接入协议[J]. J4, 2013, 40(2): 181-186.
[13]	李建东;薛富国;杨春刚;李维英;石华. 认知网络中快速自适应功率控制算法[J]. J4, 2010, 37(2): 186-191.
[14]	张天魁;程婕;冯春燕. 一种全速率的多天线中继STBC协作机制[J]. J4, 2010, 37(2): 334-339.
[15]	黑永强;李晓辉;易克初;杨宏. 一种波束成形的分组空时编码系统检测算法[J]. J4, 2009, 36(4): 649-654.