自监督对比学习的CNN-GRU语音情感识别算法

doi:10.19665/j.issn1001-2400.20241109

摘要/Abstract

摘要：

在线语音情感识别领域中训练模型的困难往往来自于极大的输入数据,捕捉语音中有意义的情感信息及满足复杂网络结构带来的高并发实时性要求变得十分困难。为此提出了一种使用自监督对比学习的CNN-GRU轻量级在线语音情感识别方法。通过对比预测编码CPC的损失函数进行模型预训练,学习长对话语音中的关键情感特征表达解决语音特征提取问题;设计轻量级CNN-GRU的编码器解码器网络结构解决在线检测的实时性需求。原始音频被输入到1D-CNN中以提取音频特征,然后音频特征被输入到GRU单元中并最终得到情感类别标签。在某外呼平台收集的数千条对话录音中进行实验对比,提出的方法达到了约96.97%的准确率。实验结果表明,方法在自然场景下长对话、实时性较强的情况下具有显著的优越性。

关键词: 自监督学习, 对比预测编码, 情感识别, CNN-GRU, 轻量级架构, 在线识别

Abstract:

In the field of online speech emotion recognition,the difficulty in training models often arises from the vast input data,making it challenging to capture meaningful emotional cues within the speech or meet high concurrent real-time demands due to complex network structures.To address this problem,a lightweight online speech emotion recognition method using self-supervised contrastive learning of CNN-GRU(Convolution Neural Network-Gated Recurrent Units) is proposed.By pretraining the model with the loss function of Contrastive Predictive Coding(CPC),it learns crucial emotional feature expressions from a lengthy conversational speech,addressing the issue of speech feature extraction.A lightweight encoder-decoder network structure of CNN-GRU is designed to meet real-time detection requirements.Raw audio is fed into a 1D-CNN to extract audio features,which are then passed through GRU units to obtain emotion category labels.Experimental comparisons conducted on thousands of recorded dialogues from a call center platform demonstrate that the proposed method achieves an accuracy of 96.97%.The results indicate the superiority of this approach in scenarios involving natural conversations and strong real-time constraints.

Key words: supervised learning, compare predictive coding, emotion recognition, CNN-GRU, lightweight structures, online identification

中图分类号:

TP181

孙志, 王冠. 自监督对比学习的CNN-GRU语音情感识别算法[J]. 西安电子科技大学学报, 2024, 51(6): 182-193.

SUN Zhi, WANG Guan. CNN-GRU speech emotion recognition algorithm for self-supervised comparative learning[J]. Journal of Xidian University, 2024, 51(6): 182-193.

图/表 11

图1

图2

图3

图4

图5

图6

图7

图8

图9

表1

表2

参考文献 26

[1]	SÖNMEZ Y Ü, VAROL A. In-Depth Investigation of Speech Emotion Recognition Studies from Past to Present -The Importance of Emotion Recognition from Speech Signal for AI-[J]. Intelligent Systems with Applications, 2024:200351.
[2]	KHAN M, GUEAIEB W, EL SADDIKA, et al. MSER:Multimodal Speech Emotion Recognition Using Cross-Attention with Deep Fusion[J]. Expert Systems with Applications, 2024:122946.
[3]	SINGH J, SAHEER L B, FAUST O. Speech Emotion Recognition Using Attention Model[J]. International Journal of Environmental Research and Public Health, 2023, 20(6):5140.
[4]	CHEN W D, XING X F, CHEN P H, et al. Vesper:A Compact and Effective Pretrained Model for Speech Emotion Recognition[J]. IEEE Transactions on Affective Computing, 2024(15):1711-1724.
[5]	ZHANG K X, WEN Q S, ZHANG C L, et al. Skip-Step Contrastive Predictive Coding for Time Series Anomaly Detection[C]// ICASSP 2024-2024 IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP). Piscataway:IEEE, 2024:7065-7069.
[6]	ZHENG C Y, SALAKHUTDINOV R, EYSENBACH B. Contrastive Difference Predictive Coding(2023)[J/OL]. [2024-02-26]. https://arxiv.org/abs/2310.20141.
[7]	LEE S, PARK T, LEE K. Soft Contrastive Learning for Time Series[C]// The Twelfth International Conference on Learning Representations(ICLR 2024). Piscataway:IEEE, 2024:1-25.
[8]	MIKOLOV T, CHEN K, CORRADO G, et al. Efficient Estimation of Word Representations in Vector Space[C]// International Conference on Learning Representations. Piscataway:IEEE, 2013:1-12.
[9]	KIROS R, ZHU Y K, SALAKHUTDINOV R R, et al. Skip-thought Vectors[C]// In Advances in Neural Information Processing Systems. San Diego: NIPS, 2015:3294-3302.
[10]	RADFORD A, JOZEFOWICZ R, SUTSKEVER I. Learning to Generate Reviews and Discovering Sentiment(2017)[J/OL]. [2024-04-06]. https://arxiv.org/abs/1704.01444.
[11]	FAN R Z, POGG M, MATTOCCIA S. Contrastive Learning for Depth Prediction[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR) Workshops. Piscataway:IEEE, 2023:3226-3237.
[12]	ZHENG R J, WANG X Y, UN Y C, et al. TACO:Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learning[C]// Advances in Neural Information Processing Systems 36(NeurIPS 2023). San Diego: NIPS, 2023:48203-48225.
[13]	ZHANG R, ISOLA P, EFROS A A. Colorful Image Colorization[C]// European Conference on Computer Vision. Berlin:Springer, 2016:649-666.
[14]	OORD A V D, LI Y Z, VINYALS O. Representation Learning with Contrastive Predictive Coding(2018)[J/OL]. [2024-02-18]. https://arxiv.org/abs/1807.03748.
[15]	WISKOTT L, SEJNOWSKI T J. Slow Feature Analysis:Unsupervised Learning of Invariances[J]. Neural Computation, 2002, 14(4):715-770.
[16]	GUTMANN M, HYVÄRINEN A. NOISE-CONTRASTIVE ESTIMATION:A New Estimation Principle for Unnormalized Statistical Models[C]// Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics. Piscataway:IEEE, 2010:297-304.
[17]	BENGIO Y, SENECAL J S. Adaptive Importance Sampling to Accelerate Training of a Neural Probabilistic Language Model[J]. IEEE Transaction Neural Networks, 2008, 19(4):713-722.
[18]	CHO K, MERRIËNBOER B V, GULCEHRE C, et al. Learning Phrase Representations Using RNN Encoder Decoder for Statistical Machine Translation (2014)[J/OL]. [2023-09-12]. https://arxiv.org/pdf/1406.1078.
[19]	OORD A V D, DIELEMAN S, ZEN H, et al. WAVENET:A Generative Model for Raw Audio (2016)[J/OL]. [2024-05-11]. https://arxiv.org/abs/1609.03499.
[20]	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is All You Need[C]// Advances in Neural Information Processing Systems 30.San Diego:NIPS, 2017:5998-6008.
[21]	ZHAO Y, LIAO X, HE X, et al. Accelerated Primal-Dual Mirror Dynamics for Centralized and Distributed Constrained Convex Optimization Problems[J]. Journal of Machine Learning Research, 2023, 24(343):1-59.
[22]	ZHAO Y, HE X, ZHOU M, et al. Accelerated Primal-Dual Projection Neurodynamic Approach with Time Scaling for Linear and Set Constrained Convex Optimization Problems[J]. IEEE/CAA Journal of Automatica Sinica, 2024, 11(6):1485-1498.
[23]	BELGHAZI M I, BARATIN A, RAJESWAR S, et al. Mine:Mutual Information Neural Estimation(2018)[J/OL]. [2023-10-28]. https://arxiv.org/pdf/1801.04062.
[24]	ZHAO J F, MAO X, CHEN L J. Speech Emotion Recognition Using Deep 1D & 2D CNN LSTM Networks[J]. Biomedical Signal Processing and Control, 2019, 47:312-323.
[25]	LIM H, PARK C, MYUNG H. RONet:Real-Time Range-only Indoor Localization via Stacked Bidirectional LSTM with Residual Attention[C]// 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems(IROS). Piscataway:IEEE, 2019:3241-3247.
[26]	BAUTISTA J L, LEE Y K, SHIN H S. Speech Emotion Recognition Based on Parallel CNN-Attention Networks with Multi-Fold Data Augmentation[J]. Electronics, 2022, 11(23):3935.

模型	准确率	精确率	召回率	F1
CNN-LSTM	94.02	98.48	96.76	97.25
CNN-Bi-LSTM	96.55	100.0	95.03	98.01
并行CNN-Bi-LSTM	95.40	99.01	96.52	98.36
并行CNN-Transformer	89.94	100.00	96.00	95.63
CNN-GRU	96.97	100.00	96.88	98.41

CPC	预训练	CNN	GRU	准确率/%	精确率/%	召回率/%	F1/%
×	√	√	√	57.92	57.42	92.90	70.97
√	×	√	√	69.77	70.32	94.45	80.62
√	√	×	√	81.62	83.23	95.56	88.97
√	√	√	×	93.47	96.13	96.38	96.25
√	√	√	√	96.97	100.0	96.88	98.41