利用混合双通路神经网络的跨模态行人重识别

doi:10.19665/j.issn1001-2400.2021.05.022

摘要/Abstract

摘要：

因为红外图像数据可以有效地弥补单一可见光数据在低照度条件下的不足,因此研究跨模态可见光—红外行人重识别将为构建一个全天候、全场景智能视频监控系统提供强有力的技术支撑。跨模态行人重识别的核心问题是构建多模态数据之间的统一共享特征表达,关键在于有效区分跨模态数据中模态共享/特有的特征信息。基于此,提出了一种基于混合双通路神经网络的跨模态行人重识别方法。该方法深入地分析了混合双通路神经网络中模态共享参数层和模态特有非共享参数层对跨模态行人重识别模型的影响,同时充分利用了不同模态数据类内特征分布的一致性约束和类间相关性系数在不同模态间的一致性约束来提升模型精度。针对所设计的整体神经网络架构,该方法采用了混合自适应学习率调整的模型训练策略来提升模型的特征学习能力。最后,通过大量实验分析验证了所提方法的有效性,并且该方法在当前可见光—红外图像行人重识别两个标准数据集(SYSU-MM01和RegDB)上都取得了领先当前主流方法的识别精度。

关键词: 行人重识别, 跨模态, 双通路神经网络, 共享特征表达

Abstract:

An infrared image can effectively make up for the shortcomings of single-modality visible-light image data under low illumination conditions.Therefore,the study of cross-modality Visible-to-Infrared person re-identification will provide a strong technical support for constructing an intelligence video surveillance system under various lighting conditions.The key for cross-modality person re-identification is to construct a unified shared feature representation among multi-modal data,which needs to effectively distinguish the modal-shared/modal-specific feature information in the cross-modal data.Based on this,this paper proposes a cross-modality person re-identification method based on a hybrid dual-channel neural network.This method deeply analyzes the influence of the parameter-shared layer and the non-shared parameter layer on the cross-modality person re-identification model in the hybrid dual-channel neural network architecture.Besides cross-entropy loss,we also use the intra-class distribution and inter-class correlation constraints in the loss function to further improve the re-identification performance.In the optimization process,we effectively utilize the adaptive learning rate adjustment strategy to improve the feature learning capability of the neural network architecture.Experimental results illustrate the effectiveness of the proposed method.Also,we obtain a superior performance on the two widely used cross-modality person re-identification benchmark datasets,SYSU-MM01 and RegDB.

Key words: person re-identification, cross-modality, two-stream neural network, shared feature representation space

中图分类号:

TP391.4

程德,郝毅,周靖宇,王楠楠,高新波. 利用混合双通路神经网络的跨模态行人重识别[J]. 西安电子科技大学学报, 2021, 48(5): 190-200.

CHENG De,HAO Yi,ZHOU Jingyu,WANG Nannan,GAO Xinbo. Cross-modality person re-identification utilizing the hybrid two-stream neural networks[J]. Journal of Xidian University, 2021, 48(5): 190-200.

图/表 9

图1

图2

表1

ResNet50神经网络结构命名及参数说明"

神经网络层名称	Conv0	Conv1_x	Conv2_x	Conv3_x	Conv4_x
ResNet50 参数说明	7×7,64, stride 2 3×3 max pool, stride 2	$1 × 1,64 3 × 3,64 1 × 1,256$ ×3	$1 × 1,64 3 × 3,64 1 × 1,256$ ×3	$1 × 1,64 3 × 3,64 1 × 1,256$ ×3	$1 × 1,64 3 × 3,64 1 × 1,256$ ×3

表1

表2

表3

图3

表4

表5

表6

参考文献 27

[1]	黄凯奇, 陈晓棠, 康运锋, 等. 智能视频监控技术综述[J]. 计算机学报, 2015, 38(6):1093-1118.
	HUANG Kaiqi, CHEN Xiaotang, KANG Yunfeng, et al. Intelligent Visual Surveillance:A Review[J]. Chinese Journal of Computers, 2015, 38(6):1093-1118.
[2]	叶钰, 王正, 梁超, 等. 多源数据行人重识别研究综述[J]. 自动化学报, 2020, 46(9):1869-1884.
	YE Yu, WANG Zheng, LIANG Chao, et al. A Survey on Multi-Source Person Re-Identification[J]. Acta Automatica Sinica, 2020, 46(9):1869-1884.
[3]	LIAO S, HU Y, ZHU X, et al. Person Re-Identification by Local Maximal Occurrence Representation and Metric Learning[C]// Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2015:2197-2206.
[4]	CHENG D, CHANG X, LIU L, et al. Discriminative Dictionary Learning with Ranking Metric Embedded for Person Re-Identification[C]// Proceedings of the 26th International Joint Conference on Artificial Intelligence.New York:ACM, 2017:964-970.
[5]	CHENG D, GONG Y, ZHOU S, et al. Person Re-Identification by Multi-Channel Parts-Based CNN with Improved Triplet Loss Function[C]// Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2016:1335-1344.
[6]	LIU J, NI B, YAN Y, et al. Pose Transferrable Person Re-Identification[C]// Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2018:4099-4108.
[7]	SHEN Y, LI H, YI S, et al. Person Re-Identification with Deep Similarity-Guided Graph Neural Network[C]// Proceedings of European Conference on Computer Vision.Heidelberg:Springer, 2018:508-526.
[8]	WEI L, ZHANG S, GAO W, et al. Person Transfer GAN to Bridge Domain Gap for Person Re-Identification[C]// Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2018:79-88.
[9]	CHOPRA S, HADSELL R, LECUN Y. Learning A Similarity Metric Discriminatively,with Application to Face Verification[C]// Proceedings of 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2005:539-546.
[10]	CHEN W, CHEN X, ZHANG J, et al. Beyond Triplet Loss:A Deep Quadruplet Network for Person Re-Identification[C]// Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2017:403-412.
[11]	WANG H, WANG Y, ZHOU Z, et al. CosFace:Large Margin Cosine Loss for Deep Face Recognition[C]// Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2018:5265-5274.
[12]	ZHENG L, BIE Z, SUN Y, et al. MARS:A Video Benchmark for Large-Scale Person Re-Identification[C]// Proceedings of European Conference on Computer Vision.Heidelberg:Springer, 2016:868-884.
[13]	ZHENG L, SHENG L, TIAN L, et al. Scalable Person Re-Identification:A Benchmark[C]// Proceedings of 2015 IEEE International Conference on Computer Vision.Piscataway:IEEE, 2015:1116-1124.
[14]	WU A, ZHENG W, YU H, et al. RGB-Infrared Cross-Modality Person Re-Identification[C]// Proceedings of 2017 IEEE International Conference on Computer Vision.Piscataway:IEEE, 2017:5390-5399.
[15]	YE Mang, LAN Xiangyuan, LI Jiawei, et al. Hierarchical Discriminative Learning for Visible Thermal Person Re-Identification[C/OL]. [2021-04-30]. https://ojs.aaai.org/index.php/AAAI/outiclel/view/12293.
[16]	HAO Y, WANG N, LI J, et al. HSME:Hypersphere Manifold Embedding for Visible Thermal Person Re-Identification[C]// Proceedings of the AAAI Conference on Artificial Intelligence.Palo Alto:AAAI, 2019:8385-8392.
[17]	CHOI S, LEE S, KIM Y, et al. Hi-CMD:Hierarchical Cross-Modality Disentanglement for Visible-Infrared Person Re-Identification[C]// Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2020:10257-10266.
[18]	LI Diangang, WEI Xing, HONG Xiaopeng, et al, Infrared-Visible Cross-Modal Person Re-Identification with an X Modality[C/OL]. [2021-04-30]. https://ojs.aaai.org/index.php/AAAI/article/view/5891.
[19]	FU C, HU Y, WU X, et al. CM-NAS:Rethinking Cross-Modality Neural Architectures for Visible-Infrared Person Re-Identification (2021)[J/OL]. [2021-01-21]. https://arxiv.org/abs/2101.08467.
[20]	HAO Y, LI J, WANG N, et al. Dual-Alignment Feature Embedding for Cross-Modality Person Re-Identification[C]// Proceedings of the 27th ACM International Conference on Multimedia.New York:ACM, 2019:57-65.
[21]	HE K, ZHANG X, REN S, et al. Deep Residual Learning for Image Recognition[C]// Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2016:770-778.
[22]	PASZKE A, GROSS S, MASSA F, et al. PyTorch:An Imperative Style,High-Performance Deep Learning Library (2019)[J/OL]. [2019-12-03]. https://arxiv.org/abs/1912.01703v1.
[23]	YE M, LAN X, WANG Z, et al. Bi-Directional Center-Constrained Top-Ranking for Visible Thermal Person Re-Identification[J]. IEEE Transactions on Information Forensics and Security, 2019, 15:407-419. doi: 10.1109/TIFS.10206
[24]	WANG G, ZHANG T, CHENG J, et al. RGB-Infrared Cross-Modality Person Re-Identification via Joint Pixel and Feature Alignment[C]// Proceedings of 2019 IEEE/CVF International Conference on Computer Vision.Piscataway:IEEE, 2019:3622-3631.
[25]	FENG Z, LAI J, XIE X. Learning Modality-Specific Representations for Visible-Infrared Person Re-Identification[J]. IEEE Transactions on Image Processing, 2019, 29:579-590. doi: 10.1109/TIP.83
[26]	YE M, LAN X, LENG Q, et al. Cross-Modality Person Re-Identification via Modality-Aware Collaborative Ensemble Learning[J]. IEEE Transactions on Image Processing, 2020, 29:9387-9399. doi: 10.1109/TIP.83
[27]	WANG Guanan, ZHANG Tianzhu, YANG Yang, et al. Cross Modality Paired-Images Generation for RGB-Infrared Person Re-Identification[C/OL]. [2021-04-30].DOI: 10.1609/aaai.v34io7.6894. doi: 10.1609/aaai.v34io7.6894

参数类型	1≤epoch<10	10≤epoch<20	20≤epoch
骨干网络参数	l_r×epoch×0.1×0.1	l_r×0.1	l_r×0.1×0.1
其他参数	l_r×epoch×0.1	l_r	l_r×0.1

网络结构	Top-1	Top-5	Top-10	Top-20	mAP
全共享^[9]	43.13	N/A	85.37	93.67	43.56
全共享*	47.70	75.41	85.67	93.40	46.10
双通路-0	48.41	77.56	87.33	93.48	47.60
双通路-1	47.76	75.78	85.09	92.66	45.78
双通路-2	49.79	79.31	88.35	94.50	47.83
双通路-3	45.23	75.68	83.64	91.69	44.21
双通路-4	46.83	75.05	84.85	92.90	45.01

优化策略	Top-1	Top-5	Top-10	Top-20	mAP
笔者所提优化策略	49.79	79.31	88.35	94.50	47.83
整个训练过程均使用固定的初始学习率	31.69	59.79	72.15	84.20	31.15
初始训练阶段改为使用固定的初始学习率	45.73	74.68	85.43	92.60	44.97
初始训练阶段改为使用固定的1/10初始学习率	38.86	69.76	80.36	88.98	37.97

方法	Top-1	Top-5	Top-10	Top-20	mAP
Zero-Pad^[14]	14.80	N/A	54.12	71.33	15.95
TONE^[15]	12.52	N/A	50.72	68.60	14.42
BDTR^[23]	17.01	N/A	55.43	71.96	19.66
HSME^[16]	20.68	N/A	62.74	77.95	23.12
AlignGAN^[24]	42.40	N/A	85.00	93.70	40.70
MSR^[25]	37.35	N/A	83.40	93.34	38.11
MACE^[26]	51.64	N/A	87.25	94.44	50.11
JSIA^[27]	38.10	N/A	80.70	89.90	36.90
DFE^[20]	48.71	N/A	88.86	95.27	48.59
DFE*^[20]	49.12	N/A	86.06	92.80	47.92
文中所提方法(双通路-0*)	51.41	77.99	86.83	94.08	49.06
文中所提方法(双通路-2*)	54.22	81.36	88.40	94.53	50.58

方法	可见光—红外
方法	Top-1	Top-10	Top-20	mAP	Top-1	Top-10	Top-20	mAP
Zero-Pad^[14]	17.74	34.21	44.35	18.90	16.63	34.68	44.25	17.82
HCML^[15]	24.44	47.53	56.78	20.08	21.70	45.02	55.58	22.24
BDTR^[23]	33.56	58.61	67.43	32.76	32.92	58.46	68.43	31.96
HSME^[16]	50.85	73.36	81.66	47.00	50.15	72.40	81.07	46.16
AlignGAN^[24]	57.9	N/A	N/A	53.6	56.3	N/A	N/A	53.4
MSR^[25]	48.43	70.32	79.95	48.67	48.43	70.32	79.95	48.67
MACE^[26]	72.37	88.40	93.59	69.09	72.12	88.07	93.07	68.57
JSIA^[27]	48.5	N/A	N/A	49.3	48.1	N/A	N/A	48.9
DFE^[20]	70.13	86.32	91.96	69.14	67.99	85.56	91.41	66.70
双通路-2*	82.54	92.60	95.96	78.80	78.51	90.75	94.51	75.51