基于关键区域特征提取与二阶段分类网络的场景识别方法

doi:10.16180/j.cnki.issn1007-7820.2024.07.004

摘要/Abstract

摘要：

在场景识别任务中,存在异类场景包含高相似度的物品种类或同类场景空间布局差异过大的情况,即场景的类间相似性与类内差异性。现有方法通过增强数据集或利用多层次的信息互补提高分类器的判别能力,尽管性能得到了一定提升,但仍存在局限性。文中提出了关键区域特征提取(Discriminative Patch Extraction,DPE)以及二阶段分类(Two-Stage Classification,TSC)网络的方法来克服场景的类间相似性与类内差异性。关键区域提取通过保留图像中的关键信息区域来避免类内差异性对场景识别的影响,而二阶段分类网络则通过粗细两个阶段的训练来避免类间相似性对场景识别的影响。文中方法结合ViT(Vision Transformer)等基线网络,在经典场景识别数据集Scene15、MITindoor67和SUN397上的分类精度分别达到了96.9%、88.4%以及76.0%。所提方法在最大规模的场景识别数据集Places365上取得了60.5%的最高分类精度。

关键词: 场景识别, 深度神经网络, 类间相似, 类内差异, 数据增强, 关键区域特征提取, 二阶段分类, ViT

Abstract:

In the scene recognition task, there are cases where heterogeneous scenes contain items with high similarity or the spatial layout of similar scenes is too different, that is, the inter-class similarity and intra-class difference of scenes.Existing methods improve the discriminant ability of classifiers by enhancing data sets or using multi-level information complementation. Although some improvements have been made, there are still limitations.In this study, the DPE(Discriminative Patch Extraction) and TSC(Two-Stage Classification) network method are proposed to overcome the inter-class similarity and intra-class difference of scenes. DPE avoids the impact of intra-class differences on scene recognition by preserving the key information regions in images, while the TSC network avoids the impact of inter-class similarities on scene recognition by the coarse-fine two-stage training.After combining the proposed method with baseline networks such as ViT(Vision Transformer), the classification accuracy of classical scene recognition data sets Scene15, MITindoor67 and SUN397 reaches 96.9%, 88.4% and 76.0%, respectively. The proposed method achieves the highest classification accuracy of 60.5% on the largest scene recognition dataset Places365.

Key words: scene recognition, deep neural networks, inter-class similarity, intra-class variability, data augmentation, discriminative patch extraction, two-stage classification, ViT

中图分类号:

TP391

韩瀛昊, 李菲菲. 基于关键区域特征提取与二阶段分类网络的场景识别方法[J]. 电子科技, 2024, 37(7): 25-32.

HAN Yinghao, LI Feifei. Scene Recognition Algorithm Based on Discriminative Patch Extraction and Two-Stage Classification[J]. Electronic Science and Technology, 2024, 37(7): 25-32.

图/表 10

图1

图2

图3

图4

图5

表1

表2

表3

表4

表5

参考文献 26

[1]	Simonyan K, Zisserman A. Very deep convolutional networks for largescale image recognition[EB/OL].(2015-04-15) [2023-02-02] https://arxiv.org/pdf/1409.1556v4.pdf.
[2]	Ioffe S, Szegedy C. Batch normalization:Accelerating deep network training by reducing internal covariate shift[C]. Guangzhou: Proceedings of the Thirty-second International Conference on International Conference on Machine Learning, 2015:448-456.
[3]	He K, Zhang X, Ren S, et al. Deep residual learning forimage recognition[C]. Las Vegas: IEEE Conference on Computer Vision and Pattern Recognition, 2016:770-778.
[4]	Deng J, Dong W, Socher R, et al. Imagenet:A large-scale hierarchical image database[C]. Miami: IEEE Conference on Computer Vision and Pattern Recognition, 2009:248-255.
[5]	Zhou B Lapedriza A, Khosla A, et al. Places:A 10 million image database for scene recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 40(6):1452-1464.
[6]	Liu Y, Chen Q, Chen W, et al. Dictionary learning inspired deep network for scene recognition[C]. New Orleans: AAAI Conference on Artificial Intelligence, 2018:7178-7185.
[7]	Yang S, Ramanan D. Multi-scale recognition with DAG-CNNs[C]. Santiago: IEEE International Conference on Computer Vision, 2015:1215-1223.
[8]	Liu S, Tian G, Xu Y. A novel scene classification model combining ResNet based transfer learning and dataaugmentation with a filter[J]. Neurocomputing, 2019, 33(8):191-206.
[9]	Cheng X, Lu J, Feng J, et al. Scene recognition with objectness[J]. Pattern Recognition, 2018, 74(2):474-487.
[10]	Chen G, Song X, Zeng H, et al. Scene recognition with prototype-agnostic scene layout[J]. IEEE Transactions on Image Processing, 2020, 29(1):5877-5888.
[11]	Seong H, Hyun J, Kim E. Fosnet:An end-to-end trainable deep neural network for scene recognition[J]. IEEE Access, 2020(8):82066-82077.
[12]	López-Cifuentes A, Escudero-Vinolo M, Bescós J, et al. Semantic-aware scene recognition[J]. Pattern Recognition, 2020, 10(2):107256-107262.
[13]	Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16×16 words:Transformers for image recognition at scale[EB/OL].(2021-06-03) [2023-02-03] https://arxiv.org/abs/2010.11929.
[14]	Wang Z, Wang L, Wang Y, et al. Weakly supervised pat-chnets:Describing and aggregating local patches for scene recognition[J]. IEEE Transactions on Image Processing, 2017, 26(4):2028-2041. doi: 10.1109/TIP.2017.2666739 pmid: 28207394
[15]	Tang P, Wang H, Kwong S. G-MS2F:Goog LeNet based multistage feature fusion of deep CNN for scene recognition[J]. Neurocomputing, 2017, 22(5):188-197.
[16]	Wang L, Guo S, Huang W, et al. Knowledge guided disambiguation for largescale scene classification with multi-resolution CNNs[J]. IEEE Transactions on Image Processing, 2017, 26(4):2055-2068.
[17]	Zeng H, Song X, Chen G, et al. Amorphous region context modeling for scene recognition[J]. IEEE Transactions on Multimedia, 2020, 24(9):141-151.
[18]	Qiu J, Yang Y, Wang X, et al. Scene essence[C]. Online: IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021:8322-8333.
[19]	谢林, 李菲菲, 陈虬. 基于稀疏自动编码机的场景识别算法[J]. 电子科技, 2019, 32(1):38-41.
	Xie Lin, Li Feifei, Chen Qiu. Scene recognition algorithm based on sparse autoencoder[J]. Electronic Science and Technology, 2019, 32(1):38-41.
[20]	Xie L, Lee F, Liu L, et al. Hierarchical coding of convolutional features for scene recognition[J]. IEEE Transactions on Multimedia, 2019, 22(5):1182-1192.
[21]	Hayat M, Khan S H, Bennamoun M, et al. A spatial layout and scale invariant feature representation for indoor scene classification[J]. IEEE Transactions on Image Processing, 2016, 25(10):4829-4841.
[22]	缪冉, 李菲菲, 陈虬. 基于卷积神经网络与多尺度空间编码的场景识别方法[J]. 电子科技, 2020, 33(12):1-7.
	Miao Ran, Li Feifei, Chen Qiu. Scene recognition algorithm based on convolutional neural networks and multi-scale space encoding[J]. Electronic Science and Technology, 2020, 33(12):1-7.
[23]	Shao X, Zhang J, Bao B K, et al. Automatic scene recognition based on constructed knowledge space learning[J]. IEEE Access, 2019(7):102902-102910.
[24]	Lim K L, Jiang X, Yi C. Deep clustering with variational autoencoder[J]. IEEE Signal Processing Letters, 2020, 27(2):231-235.
[25]	Song X, Jiang S, Herranz L. Multiscale multi-feature context modeling for scene recognition in the semantic manifold[J]. IEEE Transactions on Image Processing, 2017, 26(6):2721-2735.
[26]	He K, Chen X, Xie S, et al. Masked autoencoders are scalable vision learners[C]. New Orleans: IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022:15979-15988.

模型	准确率/%
DL-CNN^[6]	86.40
VSAD^[14]	86.10
G-MS2F^[15]	79.60
FTOTLM^[8]	74.60
SDO^[9]	86.80
LGN^[10]	88.10
MR CNN^[16]	86.70
ARG^[17]	88.13
Scene Essence^[18]	83.92
SaSR^[12]	86.10
ViT^[13]	84.20
DPE+TSC SaSR (本文)	86.70
DPE+TSC ViT (本文)	88.40

模型	准确率/%
DL-CNN^[6]	70.1
VSAD^[14]	72.0
G-MS2F^[15]	64.1
FTOTLM^[8]	65.5
SDO^[9]	73.4
LGN^[10]	74.1
MR CNN^[16]	72.0
ARG^[17]	75.0
Scene Essence^[18]	68.3
SaSR^[12]	74.0
ViT^[13]	75.0
DPE+TSC SaSR(本文)	74.7
DPE+TSC ViT (本文)	76.0

模型	准确率/%
HOG + SPM 3^[19]	88.3
DL-CNN^[6]	96.0
NNSD^[20]	94.7
G-MS2F^[16]	93.2
FTOTLM^[8]	94.0
SDO^[9]	95.9
S2ICA^[21]	93.1
DAG CNN^[7]	92.9
Multi-Scale Space VLAD^[22]	94.7
CKSL^[23]	94.4
VAED^[24]	88.1
ViT^[13]	94.7
DPE+ViT (本文)	96.4
DPE+TSC ViT (本文)	96.9

模型	准确率/%
LGN^[10]	56.5
EMFS^[25]	54.3
SaSR^[12]	56.5
Scene Essence^[18]	55.2
MAE^[26]	60.3
Places365-VGG^[5]	55.2
Places365- ResNet50^[5]	54.7
DPE+ResNet50 (本文)	54.9
DPE+TSC ResNet50 (本文)	55.1
DPE+ViT (本文)	58.5
DPE+TSC ViT (本文)	60.5

基线网络 (“√”表示选择该网络)			消融模块 (“√”表示引入该模块)		准确度/%
ResNet50	SaSR	ViT	DPE	TSC	Scene15	MIT 67	SUN397	Places365
√	-	-	-	-	92.1	84.0	70.8	54.7
√	-	-	√	-	93.7	84.7	71.1	54.9
√	-	-	-	√	94.1	84.9	71.8	54.9
√	-	-	√	√	94.6	85.5	72.3	55.1
-	√	-	-	-	94.2	86.1	73.9	56.5
-	√	-	√	-	94.6	86.4	74.1	56.6
-	√	-	-	√	95.1	86.5	74.4	56.9
-	√	-	√	√	95.5	86.7	74.7	57.2
-	-	√	-	-	94.7	84.2	75.0	57.7
-	-	√	√		96.2	87.5	75.8	58.5
-	-	√	-	√	96.4	88.0	75.8	59.1
-	-	√	√	√	96.9	88.4	76.0	60.5