Q学习差分进化算法求解热电动态经济排放调度

doi:10.16180/j.cnki.issn1007-7820.2024.05.002

摘要/Abstract

摘要：

热电联产动态经济排放调度同时考虑了燃料成本花费和污染气体排放两个目标值,且下一时间段的热电产量受当前时间段热电产量的影响,这是近年来电力系统运行中的一个重要问题。文中提出一种基于Q学习强化多目标差分进化(Q Learning Multi-Objective Differential Evolution,QLMODE)算法,以此求解热电联产动态经济排放调度(Combined Heat and Power Dynamic Economic Emission Dispatch,CHPDEED)问题。在QLMODE中,采用Q学习技术调整算法的比例因子参数,即在迭代过程中利用子代解和父代解之间的支配关系确定动作奖励和惩罚,并通过Q学习调整参数值,以获得最适合环境模型的算法参数。文中将所提QLMODE用于求解11机组和33机组的热电联产动态经济排放调度问题。仿真结果表明,与4种成熟的多目标优化算法相比,QLMODE算法燃料成本最小,污染气体排放最少,收敛性和多样性指标优于其他4种算法,且QLMODE在两组问题上都获得了更好的Pareto最优前沿。

关键词: Q学习, 强化学习, 多目标算法, 差分进化, 热电联产, 经济排放调度, 动态调度, 电力系统

Abstract:

The dynamic economic emission scheduling of cogeneration takes into account both fuel cost and pollution gas emission, and the thermoelectricity output in the next period is affected by the thermoelectricity output in the current period, which is an important problem in power system operation in recent years. In this study, a new QLMODE(Q-Learning Multi-Objective Differential Evolution) algorithm is proposed to solve the CHPDEED(Combined Heat and Power Dynamic Economic Emission Dispatch) problem. In QLMODE, the Q-learning technique is used to adjust the scale factor parameters of the algorithm, that is, in the iterative process, the action reward and punishment are determined by using the dominant relationship between the child solution and the parent solution, and the parameter values are adjusted by Q-learning to obtain the most suitable algorithm parameters for the environmental model. The proposed QLMODE is used to solve the CHPDEED with 11 units and 33 units. The simulation results show that compared with four mature multi-objective optimization algorithms, the QLMODE algorithm has the least fuel cost and the least pollution gas emission, the convergence and diversity index of QLMODE algorithm is better than the other four algorithms, and QLMODE has a better Pareto optimal frontier on both sets of problems.

Key words: Q learning, reinforcement learning, multi-objective algorithm, differential evolution, cogeneration combined heat and power, economic emission dispatch, dynamic dispatch, power system

中图分类号:

TP18

方帅, 陈旭, 李康吉. Q学习差分进化算法求解热电动态经济排放调度[J]. 电子科技, 2024, 37(5): 9-17.

FANG Shuai, CHEN Xu, LI Kangji. A Q-Learning Differential Evolution Algorithm for Combined Heat and Power Dynamic Economic Emission Dispatch[J]. Electronic Science and Technology, 2024, 37(5): 9-17.

图/表 13

图1

图2

图3

图4

表1

表2

表3

图5

表4

表5

表6

图6

表7

参考文献 28

[1]	杨德友, 刘世宇. 求解电力系统多目标环境经济调度的帕累托最优MFO算法[J]. 电工电能新技术, 2018, 37(2):30-37. doi: 10.12067/ATEEE1706064
	Yang Deyou, Liu Shiyu. Multi-objective economic-enviro-nmental dispatch based on Pareto optimal moth-flame optimization algorithm[J]. Advanced Technology of Electrical Engineering and Energy, 2018, 37(2):30-37. doi: 10.12067/ATEEE1706064
[2]	王玉梅, 张继钦, 周永鑫. 计及碳交易和价格型需求响应的热电联合低碳经济调度[J]. 电子科技, 2023, 36(10):74-81.
	Wang Yumei, Zhang Jiqin, Zhou Yongxin. Combined heat and power low-carbon economic scheduling consideringcarbon trading and price demand response[J]. Eletronic Science and Techonolgy, 2023, 36(10):74-81.
[3]	Guo T, Henwood M I, Van Ooijen M. An algorithm for combined heat and power economic dispatch[J]. IEEE Transactions on Power Systems, 1996, 11(4):1778-1784.
[4]	Narang N, Sharma E, Dhillon J S. Combined heat and power economic dispatch using integrated civilized swarm optimization and Powell's pattern search method[J]. Applied Soft Computing, 2017, 52(1):190-202.
[5]	Nasir M, Sadollah A, Aydilek Í B, et al. A combination of FA and SRPSO algorithm for combined heat and power economic dispatch[J]. Applied Soft Computing, 2021, 102(1):88-98.
[6]	Khalili M, Nikoukar J, Sedighizadeh M. Combined heat and power economic dispatch using improved shuffled frog leaping algorithm[J]. Journal of Advances in Computer Research, 2021, 12(1):1-11.
[7]	Zou D, Li S, Kong X, et al. Solving the combined heat and power economic dispatch problems by an improved genetic algorithm and a new constraint handling strategy[J]. Applied Energy, 2019, 237(1):646-670.
[8]	Sundaram A. Multiobjective multiverse optimization algorithm to solve combined economic,heat and power emission dispatch problems[J]. Applied Soft Computing, 2020, 91(1):6195-6209.
[9]	Xiong G, Shuai M, Hu X. Combined heat and power economic emission dispatch using improved bare-bone multiobjective particle swarm optimization[J]. Energy, 2022, 244(1):3108-3127.
[10]	Jadoun V K, Prashanth G R, Joshi S S, et al. Optimal fuzzy based economic emission dispatch of combined heat and power units using dynamically controlled whale optimization algorithm[J]. Applied Energy, 2022, 315(1):33-50.
[11]	Niknam T, Azizipanah-Abarghooee R, Roosta A, et al. A new multiobjective reserve constrained combined heatand power dynamic economic emission dispatch[J]. Energy, 2012, 42(1):530-545.
[12]	Elaiw A M, Xia X, Shehata A M. Combined heat and power dynamic economic dispatch with emission limitations using hybrid DE-SQP method[J]. Abstract and Applied Analysis, 2013(7):1-10.
[13]	Basu M. Combined heat and power economic emission dispatch using nondominated sorting genetic algorithm-II[J]. International Journal of Electrical Power and Energy Systems, 2013, 53(1):135-141.
[14]	Alomoush M I. Application of the stochastic fractal search algorithm and compromise programming to combined heat and power economic-emission dispatch[J]. Engineering Optimization, 2020, 52(11):1992-2010.
[15]	Chen X, Li K, Xu B, et al. Biogeography-based learning particle swarm optimization for combined heat and power economic dispatch problem[J]. Knowledge-Based Systems, 2020, 208(15):106463-106469.
[16]	赖俊, 魏竞毅, 陈希亮. 分层强化学习综述[J]. 计算机工程与应用, 2021, 57(3):72-79. doi: 10.3778/j.issn.1002-8331.2010-0038
	Lai Jun, Wei Jingyi, Chen Xiliang. Overview of hierarchical reinforcement learnin[J]. Computer Engineering and Applications, 201, 57(3):72-79.
[17]	郑敦勇, 姚宜斌, 聂文锋, 等. 基于机器学习集成算法的电离层层析算法迭代初值精化[J]. 地球物理学报, 2022, 65(8):2796-2812.
	Zheng Dunyong, Yao Yibin, Nie Wenfeng, et al. Iterative initial value refinement of ionospheric tomography algorithm based on machine learning ensemble algorithm[J]. Chinese Journal of Geophysics, 2022, 65(8):2796-2812.
[18]	Hu Z, Gong W, Li S. Reinforcement learning-based differential evolution for parameters extraction of photovoltaic models[J]. Energy Reports, 2021(7):916-928.
[19]	张秦浩, 敖百强, 张秦雪. Q-learning强化学习制导律[J]. 系统工程与电子技术, 2020, 42(2):414-419. doi: 10.3969/j.issn.1001-506X.2020.02.21
	Zhang Qinhao, Ao Baiqiang, Zhang Qinxue. Reinforcemen tlearning guidance law of Q-learning[J]. Systems Engineering and Electronics, 2020, 42(2):414-419. doi: 10.3969/j.issn.1001-506X.2020.02.21
[20]	Watkins C J C H, Dayan P. Q-learning[J]. Machine Learning, 1992, 8(3):279-292.
[21]	葛传九, 武鹏, 金俊喆, 等. 基于改进差分进化算法的光伏最大功率点跟踪[J]. 电子科技, 2022, 35(9):15-21.
	Ge Chuanjiu, Wu Peng, Jin Junzhe, et al. Photovoltaic maximum power point tracking based on improved differential evolution algorithm[J]. Electronic Science and Technology, 2022, 35(9):15-21.
[22]	冯士刚, 艾芊. 带精英策略的快速非支配排序遗传算法在多目标无功优化中的应用[J]. 电工技术学报, 2007(12):146-151.
	Feng Shigang, Ai Qian. Application of fast and elitist non-dominated sorting generic algorithm in multiobjective reactive power optimization[J]. Transaction of China Electrotechnical Society, 2007(12):146-151.
[23]	Shaabani Y A, Seifi A R, Kouhanjani M J. Stochastic multiobjective optimization of combined heat and power economic/emission dispatch[J]. Energy, 2017, 141(15):1892-1904.
[24]	Kukkonen S, Lampinen J. GDE3:The third evolution step of generalized differential evolution[C]. Edinburgh: IEEE Congress on Evolutionary Computation, 2005:117-128.
[25]	Li H, Zhang Q. Multi-objective optimization problems with complicated Pareto sets,MOEA/D and NSGA-II[J]. IEEE Transactions on Evolutionary Computation, 2008, 13(2):284-302.
[26]	Chen X, Du W, Qian F. Multi-objective differential evolution with ranking-based mutation operator and it's application in chemical process optimization[J]. Chemometrics and Intelligent Laboratory Systems, 2014, 136(15):85-96.
[27]	Zitzler E, Thiele L. Multi-objective evolutionary algorithms:A comparative case study and the strength Pareto approach[J]. IEEE Transactions on Evolutionary Computation, 1999, 3(4):257-271.
[28]	Coello C A C, Cortés N C. Solving multi-objective optimization problems using an artificial immune system[J]. Genetic Programming and Evolvable Machines, 2005, 6(2):163-190.

算法	参数
TVMOPSO	种群大小N=100,权重系数w_min=0.1,w_max=0.9, 加速度系数C₁_f=C₂_i=0.5,C₁_i=C₂_f=2
GDE3	N=100,比例因子F=0.5,交叉因子CR =0.5
NSGA-II-DE	N=100,多项式变异率η =20,F=0.5,CR =0.5
MODE-RMO	N=100,F=0.5,CR =0.5
QLMODE	N=100,CR=0.5,α =0.1,γ =0.5

时间 /h	电力需求 /MW	热量需求 /MWth	时间 /h	电力需求 /MW	热量需求 /MWth
1	1 036	390	13	2 072	470
2	1 110	400	14	1 924	460
3	1 258	410	15	1 776	450
4	1 406	420	16	1 554	450
5	1 480	440	17	1 480	420
6	1 628	450	18	1 628	435
7	1 702	450	19	1 776	445
8	1 776	455	20	1 972	450
9	1 924	460	21	1 924	445
10	2 022	460	22	1 628	435
11	2 106	470	23	1 332	400
12	2 150	480	24	1 184	400

		TV-MOPSO	GDE3	NSGA-II-DE	MODE-RMO	QLMDOE
经济调度	花费/美元	2 571 077.76	2 601 166.27	2 570 790.17	2 586 935.69	2 568 391.84
经济调度	排放/kg	268 580.93	288 674.01	279 342.01	279 073.88	268 279.16
排放调度	花费/美元	2 654 159.60	2 692 465.06	2 692 234.83	2 691 947.73	2 682 440.57
排放调度	排放/kg	250 458.36	255 900.25	248 606.82	254 109.93	246 147.21
经济排放调度	花费/美元	2 598 219.81	2 611 266.56	2 600 049.17	2 606 718.77	2 597 173.09
经济排放调度	排放/kg	256 965.00	271 013.55	260 177.24	265 514.19	256 638.51
	时间/s	884.65	549.20	584.01	888.32	660.29

指标	算法	MIN	MEAN	MAX	STD	Sig.
HV	TVMOPSO	0.024 8	0.025 5	0.026 1	0.000 3	+
	GDE3	0.021 5	0.022 4	0.022 9	0.000 4	+
	NSGA-II-DE	0.025 3	0.026 0	0.026 8	0.000 3	+
	MODE-RMO	0.022 5	0.023 1	0.024 0	0.000 4	+
	QLMODE	0.026 3	0.026 9	0.027 6	0.000 4
IGD	TVMOPSO	5 428.856 1	9 173.081 3	13 339.615 5	1 968.893 5	+
	GDE3	13 765.600 8	18 296.121 3	23 561.906 5	2 109.964 2	+
	NSGA-II-DE	3 735.038 7	5 957.392 7	8 399.362 4	1 277.406 4	+
	MODE-RMO	12 970.203 4	15 633.338 8	18 384.426 1	1 446.620 2	+
	QLMODE	1 469.630 7	3 883.327 9	6 476.832 7	1 327.309 0

时间 /h	电力需求 /MW	热量需求 /MWth	时间 /h	电力需求 /MW	热量需求 /MWth
1	3 108	1 170	13	6 216	1 410
2	3 330	1 200	14	5 772	1 380
3	3 774	1 230	15	5 328	1 350
4	4 218	1 260	16	4 662	1 350
5	4 440	1 320	17	4 440	1 260
6	4 884	1 350	18	4 884	1 305
7	5 106	1 350	19	5 328	1 335
8	5 328	1 365	20	5 916	1 350
9	5 772	1 380	21	5 772	1 335
10	6 066	1 380	22	4 884	1 305
11	6 318	1 410	23	3 996	1 200
12	6 450	1 440	24	3 552	1 200