双域渐进算法优化的自适应指导机制通道剪枝

doi:10.19665/j.issn1001-2400.20250110

摘要/Abstract

摘要：

现有的剪枝方法面临着通道选择标准不一致问题,使通道的选择趋势存在明显差异,这种差异导致了剪枝策略中出现盲点。为了解决这一问题,提出了一种基于双域渐进算法优化的自适应指导机制的通道剪枝方法。具体而言,该指导机制引入了一个正则化项来分配惩罚权重,平衡了单个标准的得分和多个标准的统一性。此外,双域渐进优化算法根据空间关系变化与迭代搜索进度动态调整搜索策略,灵活确定指导机制的最优作用深度和广度,达到最佳剪枝效果。所提方法以最优的作用深度和广度协调了基于个体视角与整体视角剪枝标准之间的对立和统一,形成了全面而高效的剪枝体系。实验结果表明,所提出的剪枝方法优于现有的剪枝方法,以最小的精度损失显著降低了模型参数规模和运算量。在CIFAR-10数据集上,所提方法将VGG-16模型压缩至原始大小的11.15%,而精度仅下降0.04%。在ImageNet数据集上,将ResNet-50模型的参数减少至27.01%,同时保持了73.83%的准确率。

关键词: 深度学习, 卷积神经网络, 通道剪枝, 模型压缩

Abstract:

Current pruning methods often face inconsistencies in channel selection criteria,resulting in noticeable disparities in channel selection trends.This discrepancy contributes to the emergence of blind spots within pruning strategies.To address this issue,we propose a channel pruning method via an Adaptive Guidance Mechanism (AGM) optimized by the Dual-Domain Progressive Optimization Algorithm.Specifically,the Guiding Mechanism introduces a regularization term to allocate penalty levels,balancing individual criterion scores with the convergence of multiple criteria.In addition,the Dual-Domain Progressive Optimization Algorithm dynamically adjusts the search strategy based on changes in spatial relationships and the progress of iterative search,flexibly determining the optimal depth and breadth of the guidance mechanism to achieve the best pruning performance.The proposed AGM harmonizes the opposition and unity between the individual perspective-based and overall perspective-based pruning criteria with the optimal depth and breadth of impact,forming a cohesive and comprehensive pruning system.Experimental results demonstrate that the proposed pruning method outperforms existing pruning methods,significantly reducing model parameters and FLOPs with a minimal accuracy loss.For example,it compresses the VGG-16 network on CIFAR-10 to 11.15% of its original size with just a 0.04% accuracy drop.On ImageNet,it reduces ResNet-50 parameters to 27.01% while maintaining a 73.83% accuracy.

Key words: deep learning, convolutional neural network, channel pruning, model compression

中图分类号:

TP391

冯进, 郭杰, 张铭津, 李云松, 徐志远. 双域渐进算法优化的自适应指导机制通道剪枝[J]. 西安电子科技大学学报, 2025, 52(3): 202-216.

FENG Jin, GUO Jie, ZHANG Mingjin, LI Yunsong, XU Zhiyuan. Channel pruning via an adaptive guidance mechanism optimized by the dual-domain progressive optimization algorithm[J]. Journal of Xidian University, 2025, 52(3): 202-216.

图/表 13

图1

图2

表1

表2

表3

表4

表5

图3

表6

表7

表8

表9

ImageNet测试集上的ResNet-50网络"

方法	原模型 Top-1 准确率/%	剪枝后 Top-1 准确率/%	变化的 Top-1 准确率/%	原模型 FLOPs/	剪枝后 FLOPs/	FLOPs 减少/%	原模型参数量	剪枝后参数量	参数减少量/%
DECORE^[18]	76.15	76.31	+0.16	4 089.12	3 539.13	13.45	25.56	22.74	11.02
GKP-TMI^[23]	76.15	75.96	-0.19	4 089.01	3 168.98	22.50	25.56	19.91	22.10
SOSP^[30]	76.15	76.60	+0.45	4 089.00	2 944.08	28.00	25.56	17.89	30.00
CLR-RNF^[27]	76.01	74.85	-1.16	4 089.05	2 437.48	40.39	25.56	16.92	33.80
MFP^[9]	76.15	75.67	-0.48	4 089.00	2 363.44	42.20
CCEP^[31]	76.13	76.06	-0.07	4 089.00	2 266.94	44.56
ℓ1-norm^[7]	76.15	75.18	-0.97	4 110.22	2 270.07	44.77	25.55	15.09	40.80
CHIP^[8]	76.15	76.41	+0.26	4 110.22	2 257.13	44.80	25.55	15.09	40.80
AGM	76.15	76.63	+0.48	4 110.22	2 202.78	46.40	25.55	14.79	42.11
SOSP^[30]	76.15	75.21	-0.94	4 089.00	2 248.95	45.00	25.56	13.04	49.00
SPWB^[32]	76.13	75.62	-0.51	4 102.04	2 010.00	51.00	25.57	12.94	49.40
GNN-RL^[29]	76.10	74.28	-1.82	4 089.00	1 921.83	53.00
MFP^[9]	76.15	74.86	-1.29	4 088.99	1 901.38	53.50
MSVFP^[11]	76.64	75.53	-1.11	4 088.99	1 901.38	53.50
LAASP^[10]	76.48	75.44	-1.04	4 089.00	1 885.03	53.90
ResPrune^[12]	76.15	75.10	-1.05	4 089.00	1 679.76	58.92
LFPC^[14]	76.15	74.46	-1.69	4 089.01	1 602.89	60.80
OTOv2^[20]	76.15	75.20	-0.95	4 089.01	1 525.20	62.70	25.52	12.66	50.40
ℓ1-norm^[7]	76.15	74.36	-1.79	4 110.22	1 531.06	62.75	25.55	11.05	56.70
CHIP^[8]	76.15	75.26	-0.89	4 110.22	1 521.11	62.80	25.55	11.05	56.70
AGM	76.15	75.68	-0.47	4 110.22	1 512.15	63.21	25.55	10.94	57.18
CLR-RNF^[27]	76.01	73.34	-2.67	4 089.05	1 223.72	70.07	25.56	9.00	64.79
DECORE^[18]	76.15	69.71	-6.44	4 089.12	1 189.71	70.90	25.56	6.13	76.00
DFPC^[33]	76.15	73.80	-2.34	4 089.00	1 181.72	71.10	25.54	9.64	62.26
ℓ1-norm^[7]	76.15	72.38	-3.77	4 110.22	1 002.07	75.62	25.55	8.03	68.60
CHIP^[8]	76.15	73.30	-2.85	4 110.22	952.74	76.70	25.55	8.03	68.60
OTOv2^[20]	76.15	72.20	-3.95	4 089.00	817.80	80.00	25.56	9.56	62.60
AGM	76.15	73.83	-2.32	4 110.22	773.67	81.18	25.55	6.90	72.99

表9

图4

参考文献 33

[1]	杨静雅, 齐彦丽, 周一青, 等. CNN-Transformer轻量级智能调制识别算法[J]. 西安电子科技大学学报, 2023, 50(3):40-49.
	YANG Jingya, QI Yanli, ZHOU Yiqing, et al. Algorithm for Recognition of Lightweight Intelligent Modulation Based on the CNN-Transformer Networks[J]. Journal of Xidian University, 2023, 50(3):40-49.
[2]	衡红军, 喻龙威. 基于多尺度特征信息融合的时间序列异常检测[J]. 西安电子科技大学学报, 2024, 51(3):203-214.
	HENG Hongjun, YU Longwei. Time Series Anomaly Detection Based on Multi-Scale Feature Information Fusion[J]. Journal of Xidian University, 2024, 51(3):203-214.
[3]	ZHU L, FAN H, LUO Y, et al. Temporal Cross-Layer Correlation Mining for Action Recognition[J]. IEEE Transactions on Multimedia, 2021,24:668-676.
[4]	龚峻扬, 付卫红, 方厚章. SAR图像舰船目标检测的轻量化和特征增强研究[J]. 西安电子科技大学学报, 2024, 51(2):96-106.
	GONG Junyang, FU Weihong, FANG Houzhang. Research on Lightweight and Feature Enhancement of SAR Image Ship Targets Detection[J]. Journal of Xidian University, 2024, 51(2):96-106.
[5]	张铭津, 周楠, 李云松. 平滑交互式压缩网络的红外小目标检测算法[J]. 西安电子科技大学学报, 2024, 51(4):1-14.
	ZHANG Mingjin, ZHOU Nan, LI Yunsong. Smooth Interactive Compression Network for Infrared Small Target Detection[J]. Journal of Xidian University, 2024, 51(4):1-14.
[6]	HAN S, POOL J, TRAN J, et al. Learning both Weights and Connections for Efficient Neural Network[J]. Advances in Neural Information Processing Systems, 2015,28:1-9.
[7]	LI H, KADAV A, DURDANOVIC I, et al. Pruning Filters for Efficient Convnets (2016)[J/OL]. [2017-03-10]. https://arxiv.org/abs/1608.08710.
[8]	SUI Y, YIN M, XIE Y, et al. Chip:Channel Independence-Based Pruning for Compact Neural Networks[J]. Advances in Neural Information Processing Systems, 2021,34:24604-24616.
[9]	HE Y, LIU P, ZHU L, et al. Filter Pruning by Switching to Neighboring CNNs with Good Attributes[J]. IEEE Transactions on Neural Networks and Learning Systems, 2022, 34(10):8044-8056.
[10]	GHIMIRE D, LEE K, KIM S. Loss-Aware Automatic Selection of Structured Pruning Criteria for Deep Neural Network Acceleration[J]. Image and Vision Computing, 2023,136:104745.
[11]	GHIMIRE D, KIM S H. Magnitude and Similarity Based Variable Rate Filter Pruning for Efficient Convolution Neural Networks[J]. Applied Sciences, 2022, 13(1):316.
[12]	JAYASIMHAN A, PABITHA P. ResPrune:An Energy-Efficient Restorative Filter Pruning Method Using Stochastic Optimization for Accelerating CNN[J]. Pattern Recognition, 2024,155:110671.
[13]	CHEN X, LIU C,HUP, et al. Evolving Filter Criteria for Randomly Initialized Network Pruning in Image Classification[J]. Neurocomputing, 2024,594:127872.
[14]	HE Y, DING Y, LIU P, et al. Learning Filter Pruning Criteria for Deep Convolutional Neural Networks Acceleration[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2020:2009-2018.
[15]	LIN M, JI R, WANG Y, et al. Hrank:Filter Pruning Using High-Rank Feature Map[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2020:1529-1538.
[16]	TIAN G, SUN Y, LIU Y, et al. Adding Before Pruning:Sparse Filter Fusion for Deep Convolutional Neural Networks via Auxiliary Attention[J]. IEEE Transactions on Neural Networks and Learning Systems,2021:1-13.
[17]	JIANG D, CAO Y, YANG Q. On the Channel Pruning Using Graph Convolution Network for Convolutional Neural Network Acceleration[C]// Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence (IJCAI-22). New York: IJCAI,2022:3107-3113.
[18]	ALWANI M, MADHAVAN V, WANG Y. DECORE: Deep Compression with Reinforcement Learning[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)(2021). Piscataway:IEEE,2021:12339-12349.
[19]	GUO S, ZHANG L, ZHENG X, et al. Automatic Network Pruning via Hilbert-Schmidt Independence Criterion Lasso Under Information Bottleneck Principle[C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Piscataway:IEEE,2023:17458-17469.
[20]	CHEN T, LIANG L, DING T, et al. Otov2:Automatic,Generic,User-Friendly (2023)[J/OL]. [2023-06-23]. https://arxiv.org/abs/2303.06862.
[21]	CHEN D, LIU N, ZHU Y, et al. EPSD:Early Pruning with Self-Distillation for Efficient Model Compression[C]// Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto: AAAI,2024:11258-11266.
[22]	YU S, YAO Z, GHOLAMI A, et al. Hessian-Aware Pruning and Optimal Neural Implant[C]// Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. Piscataway:IEEE,2022:3880-3891.
[23]	ZHONG S, ZHANG G, HUANG N, et al. Revisit Kernel Pruning with Lottery Regulated Grouped Convolutions[C]// International Conference on Learning Representations. La Jolla: ICLR,2021:1-12.
[24]	PARK M, KIM D, PARK C, et al. REPrune:Channel Pruning via Kernel Representative Selection[C]// Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto: AAAI,2024:14545-14553.
[25]	GAO S, LI J, ZHANG Z, et al. Device-Wise Federated Network Pruning[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2024:12342-12352.
[26]	ZHANG Y, LIN M, LIN C W, et al. Carrying out CNN Channel Pruning in a White Box[J]. IEEE Transactions on Neural Networks and Learning Systems, 2023, 34(10):7946-7955.
[27]	GUO Y, YUAN H, TAN J, et al. GDP:Stabilized Neural Network Pruning via Gates with Differentiable Polarization[C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Piscataway:IEEE,2021:5239-5250.
[28]	LIN M, JI R, LI S, et al. Network Pruning Using Adaptive Exemplar Filters[J]. IEEE Transactions on Neural Networks and Learning Systems, 2021, 33(12):7357-7366.
[29]	YU S, MAZAHERI A, JANNESARI A. Topology-Aware Network Pruning Using Multi-Stage Graph Embedding and Reinforcement Learning[C]// International Conference on Machine Learning. New York: PMLR,2022:25656-25667.
[30]	NONNENMACHER M, PFEIL T, STEINWART I, et al. SOSP:Efficiently Capturing Global Correlations by Second-Order Structured Pruning (2021)[J/OL]. [2022-06-30]. https://arxiv.org/abs/2110.11395.
[31]	SHANG H, WU J L, HONG W, et al. Neural Network Pruning by Cooperative Coevolution (2022)[J/OL]. [2022-05-09]. https://arxiv.org/abs/2204.05639.
[32]	AGARWAL P, MATHEW M, PATEL K R, et al. Prune Efficiently by Soft Pruning[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2024:2210-2217.
[33]	NARSHANA T, MURTI C, BHATTACHARYYA C. DFPC:Data Flow Driven Pruning of Coupled Channels without Data[C]// The Eleventh International Conference on Learning Representations. La Jolla: ICLR,2022:1-25.

卷积层名称	原始通道数	剪枝率
		0.53		0.55		0.62
		单层剪枝率	剪枝后通道数	单层剪枝率	剪枝后通道数	单层剪枝率	剪枝后通道数
conv0 conv1	64	0.26	47	0.35	41	0.5	32
conv3 conv4	128	0.26	94	0.35	83	0.5	64
conv6 conv7 conv8	256	0.26	189	0.35	166	0.5	128
conv10 conv11 conv12 conv14 conv15	512	0.75	128	0.75	128	0.8	102

组别	卷积层序号	原始通道数	剪枝率
			0.27		0.43
			单层剪枝率 (conv1/conv2)	剪枝后通道数 (conv1/conv2)	单层剪枝率 (conv1/conv2)	剪枝后通道数 (conv1/conv2)
第一组	2~19	16	0.44/0.15	8/13	0.5/0.5	8/8
第二组	20~37	32	0.44/0.15	17/27	0.6/0.5	12/16
第三组	38~55	64	0.44/0.00	35/64	0.7/0.0	19/64

组别	卷积层序号	原始通道数	剪枝率
			0.24		0.28		0.42
			单层剪枝率 (conv1/ conv2)	剪枝后通道数 (conv1/ conv2)	单层剪枝率 (conv1/ conv2)	剪枝后通道数 (conv1/ conv2)	单层剪枝率 (conv1/ conv2)	剪枝后通道数 (conv1/ conv2)
第一组	2~37	16	0.35/0.25	10/12	0.40/0.25	9/12	0.55/0.45	7/8
第二组	38~73	32	0.35/0.25	20/24	0.45/0.25	17/24	0.65/0.45	11/17
第三组	74~109	64	0.35/0.00	41/64	0.45/0.00	35/64	0.65/0.00	22/64

组别	卷积层序号			剪枝率
				0.16				0.25				0.45
				单层剪枝率 (conv1-2/ conv3)		剪枝后通道数 (conv1-2/ conv3)		单层剪枝率 (conv1-2/ conv3)		剪枝后通道数 (conv1-2/ conv3)		单层剪枝率 (conv1-2/ conv3)		剪枝后通道数 (conv1-2/ conv3)
第一组	2~10	256	0.35/0.15		166/217		0.5/0.27		128/186		0.6/0.7		102/76
第二组	11~22	512	0.35/0.15		332/435		0.5/0.27		256/373		0.6/0.7		204/153
第三组	23~40	1 024	0.35/0.15		665/870		0.5/0.27		512/747		0.6/0.7		409/307
第四组	41~49	2 048	0.35/0.00		1 331/2 048		0.5/0.00		1 024/2 048		0.6/0.0		819/2 048

方法	原模型 Top-1 准确率/%	剪枝后 Top-1 准确率/%	变化的 Top-1 准确率/%	原模型 FLOPs/M	剪枝后 FLOPs/M	FLOPs 减少/%	原模型参数量	剪枝后参数量	参数减少量/%
MSVFP^[11]	93.79	93.84	+0.05	313.70	154.65	50.70
HRank^[15]	93.96	93.43	-0.53	313.14	145.61	53.50	14.68	2.51	82.90
ABP^[16]	93.96	93.75	-0.21	314.60	146.19	53.53	14.75	2.44	83.46
FTWT^[17]	93.82	93.73	-0.09	314.60	138.42	56.00
ResPrune^[12]	93.76	93.49	-0.27	313.70	134.01	57.28
ℓ1-norm^[7]	93.96	93.39	-0.57	313.73	131.37	58.04	14.98	2.76	81.60
CHIP^[8]	93.96	94.02	+0.06	313.73	131.17	58.10	14.98	2.76	81.60
LAASP^[10]	93.79	93.79	+0.00	313.70	123.91	60.50
EvoFC^[13]	93.25	92.90	-0.35	313.70	119.83	61.80	14.98	4.00	73.30
AGM	93.96	94.28	+0.32	313.70	116.35	62.91	14.98	2.62	82.51
DECORE^[18]	93.96	93.56	-0.40	314.62	110.81	64.78	14.71	1.63	88.92
FTWT^[17]	93.82	93.55	-0.27	314.60	110.11	65.00
HRank^[15]	93.96	92.34	-1.62	313.14	108.91	65.38	14.68	2.60	82.38
APIB^[19]	93.96	94.00	+0.04	313.70	106.66	66.00	14.73	3.24	78.00
ABP^[16]	93.96	93.50	-0.46	314.60	106.59	66.12	14.75	2.66	81.96
ℓ1-norm^[7]	93.96	93.20	-0.76	313.73	104.96	66.54	14.98	2.51	83.30
CHIP^[8]	93.96	93.80	-0.16	313.73	104.78	66.60	14.98	2.50	83.30
AGM	93.96	94.22	+0.26	313.70	92.28	70.59	14.98	2.37	84.18
FTWT^[17]	93.82	93.19	-0.63	314.60	84.94	73.00
OTOv2^[20]	93.96	93.20	-0.76	312.87	74.15	76.30	14.90	0.73	95.10
ℓ1-norm^[7]	93.96	92.98	-0.98	313.73	67.09	78.55	14.98	1.90	87.30
CHIP^[8]	93.96	93.42	-0.54	313.73	66.95	78.60	14.98	1.90	87.30
EPSD^[21]	93.88	93.82	-0.06	312.85	62.57	80.00
AGM	93.96	93.92	-0.04	313.70	56.37	82.03	14.98	1.67	88.85