应用注意力机制的文档图像篡改与脱敏定位

doi:10.19665/j.issn1001-2400.20230105

摘要/Abstract

摘要：

诸如合同、证明文件和通知书等一些重要的文档材料,常常以电子图像格式被存储和传播。然而,由于包含关键的文字信息,此类图像往往容易被非法篡改利用,造成严重的社会影响和危害;与此同时,考虑到个人的隐私安全问题,人们往往也会对这类图像做脱除敏感信息处理。恶意篡改与脱敏均会给原始图像引入额外痕迹,但在动机上存在区别,且在操作方式上也存在一定差异。因此,有必要对二者进行区分,从而更准确地定位出篡改区域。针对这个问题,提出了一个卷积编解码网络,通过U形连接获取编码器多级特征,有效学习篡改和脱敏处理痕迹;同时,在解码网络引入多个挤压激励注意力机制模块,抑制图像内容,关注更微弱的处理痕迹,提高网络的检测能力。为了有效地辅助网络训练,构建了一个包含常见篡改操作和脱敏操作的文档图像取证数据集。实验结果表明,算法模型在此数据集上表现良好,在公开的篡改数据集上也有不错的性能,并优于对比算法。同时,所提的算法对几种常见的后处理操作具有较好的鲁棒性。

关键词: 文档图像, 篡改定位, 脱敏定位, U-Net, 挤压激励注意力机制

Abstract:

Some important documents such as contracts,certificates and notifications are often stored and disseminated in a digital format.However,due to the inclusion of key text information,such images are often easily illegally tampered with and used,causing serious social impact and harm.Meanwhile,taking personal privacy and security into account,people also tend to remove sensitive information from these digital documents.Malicious tampering and desensitization can both introduce extra traces to the original images,but there are differences in motivation and operations.Therefore,it is necessary to differentiate them to locate the tamper areas more accurately.To address this issue,we propose a convolutional encoder-decoder network,which has multi-level features of the encoder through U-Net connection,effectively learning tampering and desensitization traces.At the same time,several Squeeze-and-Excitation attention mechanism modules are introduced in the decoder to suppress image content and focus on weaker operation traces,to improve the detection ability of the network.To effectively assist network training,we build a document image forensics dataset containing common tampering and desensitization operations.Experimental results show that our model performs effectively both on this dataset and on the public tamper datasets,and outperforms comparison algorithms.At the same time,the proposed method is robust to several common post-processing operations.

Key words: document image, forgery localization, desensitization localization, U-Net, squeeze-and-excitation attention mechanism

中图分类号:

TP391

郑铿涛, 李斌, 曾锦华. 应用注意力机制的文档图像篡改与脱敏定位[J]. 西安电子科技大学学报, 2023, 50(6): 207-218.

ZHENG Kengtao, LI Bin, ZENG Jinhua. Document image forgery localization and desensitization localization using the attention mechanism[J]. Journal of Xidian University, 2023, 50(6): 207-218.

图/表 9

图1

图2

图3

表1

图4

表2

表3

表4

表5

参考文献 35

[1]	孔月萍, 刘楚, 朱旭东. 一种利用背景光流特征的虚假人脸检测方法[J]. 西安电子科技大学学报, 2021, 48(5):86-91.
	KONG Yueping, LIU Chu, ZHU Xudong. Faceanti-Spoofing Method Using the Optical Flow Features of Back Ground[J]. Journal of Xidian University, 2021, 48(5):86-91.
[2]	KIM M, TARIQ S, WOO S S. Fretal:Generalizing Deepfake Detection Using Knowledge Distillation and Representation Learning[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2021:1001-1012.
[3]	李昊东, 庄培裕, 李斌. 基于深度学习的数字图像篡改定位方法综述[J]. 信号处理, 2021, 37(12):2278-2301.
	LI Haodong, ZHUANG Peiyu, LI Bin. A Survey on Deep Learning Based Digital Image Tampering Localization Methods[J]. Journal of Signal Processing, 2021, 37(12):2278-2301.
[4]	朱叶, 余宜林, 郭迎春. HRDA-Net:面向真实场景的图像多篡改检测与定位算法[J]. 通信学报, 2022, 43(1):217-226. doi: 10.11959/j.issn.1000-436x.2022016
	ZHU Ye, YU Yilin, GUO Yingchun. HRDA-Net:Image Multiple Manipulation Detection and Location Algorithm in Real Scene[J]. Journal on Communications, 2022, 43(1):217-226. doi: 10.11959/j.issn.1000-436x.2022016
[5]	MAHDIAN B, SAIC S. Using Noise Inconsistencies for Blind Image Forensics[J]. Image and Vision Computing, 2009, 27(10):1497-1503. doi: 10.1016/j.imavis.2009.02.001
[6]	张旭, 胡晰远, 陈晨, 等. 基于透视投影下空间光照一致性分析的图像拼接篡改检测[J]. 自动化学报, 2019, 45(10):1857-1869.
	ZHANG Xu, HU Xiyuan, CHEN Chen, et al. Image Splicing Detection Based on Spatial Lighting Consistency Analysis Under Perspective Projection[J]. Acta Automatica Sinica, 2019, 45(10):1857-1869.
[7]	FERRARA P, BIANCHI T, DE ROSA A, et al. Image Forgery Localization via Fine-Grained Analysis of CFA Artifacts[J]. IEEE Transactions on Information Forensics and Security, 2012, 7(5):1566-1577. doi: 10.1109/TIFS.2012.2202227
[8]	LIN Z, HE J, TANG X, et al. Fast,Automatic and Fine-Grained Tampered JPEG Image Detection via DCT Coefficient Analysis[J]. Pattern Recognition, 2009, 42(11):2492-2501. doi: 10.1016/j.patcog.2009.03.019
[9]	BAYAR B, STAMM M C. Constrained Convolutional Neural Networks:A New Approach towards General Purpose Image Manipulation Detection[J]. IEEE Transactions on Information Forensics and Security, 2018, 13(11):2691-2706. doi: 10.1109/TIFS.2018.2825953
[10]	WU Y, ABDALMAGEED W, NATARAJAN P. Mantra-Net:Manipulation Tracing Network for Detection and Localization of Image Forgeries with Anomalous Features[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2019:9543-9552.
[11]	LONG J, SHELHAMER E, DARREL T. Fully Convolutional Networks for Semantic Segmentation[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2015:3431-3440.
[12]	吴韵清, 吴鹏, 陈北京, 等. 基于残差全卷积网络的图像拼接定位算法[J]. 应用科学学报, 2019, 37(5):651-662.
	WU Yunqing, WU Peng, CHEN Beijing, et al. Image Splicing Localization Method Based on Fully Convolutional Residual Networks[J]. Journal of Applied Sciences, 2019, 37(5):651-662.
[13]	KWON M J, YU I J, NAM S H, et al. CAT-Net:Compression Artifact Tracing Network for Detection and Localization of Image Splicing[C]// Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision.Piscataway:IEEE, 2021:375-384.
[14]	ZHUANG P, LI H, TAN S, et al. Image Tampering Localization Using a Dense Fully Convolutional Network[J]. IEEE Transactions on Information Forensics and Security, 2021, 16:2986-2999. doi: 10.1109/TIFS.2021.3070444
[15]	RONNEBERGER O, FISCHER P, BROX T. U-Net:Convolutional Networks for Biomedical ImageSegmentation[C]// International Conference on Medical Image Computing and Computer-Assisted Intervention.Heidelberg:Springer, 2015:234-241.
[16]	BI X, WEI Y, XIAO B, et al. RRU-Net:The Ringed Residual U-Net for Image Splicing Forgery Detection[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops.Piscataway:IEEE, 2019:30-39.
[17]	ZHANG R, NI J. A Dense U-Net with Cross-Layer Intersection for Detection and Localization of Image Forgery[C]// ICASSP 2020-2020 IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP).Piscataway:IEEE, 2020:2982-2986.
[18]	WU H, ZHOU J, TIAN J, et al. Robust Image Forgery Detection against Transmission over Online Social Networks[J]. IEEE Transactions on Information Forensics and Security, 2022, 17:443-456. doi: 10.1109/TIFS.2022.3144878
[19]	CRUZ F, SIDERE N, COUSTATY M, et al. Local Binary Patterns for Document Forgery Detection[C]// 2017 14th IAPR International Conference on Document Analysis and Recognition(ICDAR).Piscataway:IEEE, 2017, 1:1223-1228.
[20]	SHIVAKUMARA P, BASAVARAJA V, GOWDA H S, et al. A New RGB Based Fusion for Forged IMEI Number Detection in Mobile Images[C]// 2018 16th International Conference on Frontiers in Handwriting Recognition(ICFHR).Piscataway:IEEE, 2018:386-391.
[21]	VAN BEUSEKOM J, SHAFAIT F, BREUEL T M. Document Inspection Using Text-Line Alignment[C]// Proceedings of the 9th IAPR International Workshop on Document Analysis Systems. New York: ACM, 2010:263-270.
[22]	VAN BEUSEKOM J, SHAFAIT F, BREUEL T M. Text-Line Examination for Document Forgery Detection[J]. International Journal on Document Analysis and Recognition(IJDAR), 2013, 16(2):189-207.
[23]	BERTRAND R, GOMEZ-KRÄMER P, TERRADES O R, et al. A System Based on Intrinsic Features for Fraudulent Document Detection[C]// 2013 12th International Conference on Document Analysis and Recognition.Piscataway:IEEE, 2013:106-110.
[24]	BERTRAND R, TERRADES O R, GOMEZ-KRÄMER P, et al. A Conditional Random Field Model for Font Forgery Detection[C]// 2015 13th International Conference on Document Analysis and Recognition(ICDAR).Piscataway:IEEE, 2015:576-580.
[25]	SHANG S, KONG X, YOU X. Document Forgery Detection Using Distortion Mutation of Geometric Parameters in Characters[J]. Journal of Electronic Imaging, 2015, 24(2):023008. doi: 10.1117/1.JEI.24.2.023008
[26]	NANDANWAR L, SHIVAKUMARA P, PAL U, et al. A New Method for Detecting Altered Text in Document Images[J]. International Journal of Pattern Recognition and Artificial Intelligence, 2021, 35(12):2160010. doi: 10.1142/S0218001421600107
[27]	JAMES H, GUPTA O, RAVIV D. Learning Document Graphs with Attention for Image Manipulation Detection[C]// International Conference on Pattern Recognition and Artificial Intelligence.Heidelberg:Springer, 2022:263-274.
[28]	XU W, LUO J, ZHU C, et al. Document Images Forgery Localization Using a Two-Stream Network[J]. International Journal of Intelligent Systems, 2022, 37(8):5272-5289. doi: 10.1002/int.v37.8
[29]	LIU Z, MAO H, WU C Y, et al. A Convnet for the 2020s[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2022:11976-11986.
[30]	HU J, SHEN L, SUN G. Squeeze-and-Excitation Networks[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2018:7132-7141.
[31]	LIN T Y, GOYAL P, GIRSHICK R, et al. Focal Loss for Dense Object Detection[C]// Proceedings of the IEEE International Conference on Computer Vision.Piscataway:IEEE, 2017:2980-2988.
[32]	BERMAN M, TRIKI A R, BLASCHKO M B. The Lovász-Softmax Loss:A Tractable Surrogate for the Optimization of the Intersection-over-Union Measure in Neural Networks[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2018:4413-4421.
[33]	COZZOLINO D, VERDOLIVA L. Noiseprint:A CNN-Based Camera Model Fingerprint[J]. IEEE Transactions on Information Forensics and Security, 2019, 15:144-159. doi: 10.1109/TIFS.10206
[34]	CHEN X, DONG C, JI J, et al. Image Manipulation Detection by Multi-View Multi-Scale Supervision[C]// Proceedings of the IEEE/CVF International Conference on Computer Vision.Piscataway:IEEE, 2021:14185-14193.
[35]	ALIBABA SECURITY, CHINA SOCIETY OF IMAGE AND GRAPHICS(CSIG). Real-World Image Forgery Localization Challenge(2022) [DB/OL].[2022-10-20]. https://tianchi.aliyun.com/competition/entrance/531945/introduction.

方法	F1	IoU	MCC	AUC
ADQ1	0.153	0.098	0.152	0.650
Noiseprint	0.131	0.083	0.166	0.600
RRU-Net	0.720	0.589	0.731	0.985
MVSS-Net	0.518	0.382	0.554	0.943
DenseFCN	0.804	0.703	0.810	0.984
DIDNet	0.619	0.463	0.654	0.984
文中算法	0.832	0.726	0.832	0.955

方法	F1	IoU	MCC	AUC
RRU-Net	0.581/0.919	0.459/0.860	0.598/0.920	0.954/0.998
MVSS-Net	0.557/0.905	0.423/0.838	0.577/0.907	0.890/0.989
DenseFCN	0.519/0.811	0.388/0.715	0.540/0.823	0.945/0.997
文中算法	0.688/0.914	0.553/0.848	0.695/0.914	0.918/0.989

方法	F1	IoU	MCC	AUC
ConvNeXt-Up	0.385/0.804	0.291/0.697	0.404/0.807	0.948/0.999
U-ConvNeXt	0.463/0.882	0.346/0.797	0.486/0.882	0.912/0.997
ConvNeXt-SE	0.404/0.813	0.307/0.706	0.418/0.816	0.929/0.999
文中算法	0.688/0.914	0.553/0.848	0.695/0.914	0.918/0.989

	方法	F1	IoU	MCC	AUC
	ADQ1	0.136	0.080	0.093	0.611
	Noiseprint	0.050	0.030	0.034	0.545
	RRU-Net	0.137	0.084	0.116	0.727
直接测试	MVSS-Net	0.163	0.100	0.129	0.626
	DenseFCN	0.093	0.057	0.073	0.615
	DIDNet	0.080	0.046	0.035	0.558
	文中算法	0.211	0.143	0.225	0.679
	RRU-Net	0.311(+0.174)	0.217(+0.133)	0.304(+0.188)	0.768(+0.041)
	MVSS-Net	0.288(+0.125)	0.193(+0.093)	0.281(+0.152)	0.716(+0.090)
微调	DenseFCN	0.197(+0.104)	0.137(+0.080)	0.200(+0.127)	0.798(+0.183)
	DIDNet	0.135(+0.055)	0.081(+0.035)	0.107(+0.072)	0.696(+0.138)
	文中算法	0.402(+0.191)	0.305(+0.162)	0.409(+0.184)	0.864(+0.185)

后处理操作	参数	F1	IoU	MCC	AUC
	3×3	0.659/0.910	0.525/0.842	0.668/0.910	0.906/0.987
MB	5×5	0.583/0.864	0.450/0.781	0.594/0.868	0.882/0.982
	7×7	0.450/0.785	0.332/0.687	0.463/0.797	0.839/0.977
	5	0.541/0.742	0.414/0.628	0.552/0.763	0.846/0.899
GN	10	0.430/0.567	0.316/0.462	0.442/0.603	0.796/0.817
	15	0.362/0.463	0.259/0.375	0.375/0.500	0.758/0.764
	1.5	0.643/0.908	0.502/0.838	0.652/0.908	0.861/0.991
RS	0.5	0.589/0.899	0.451/0.824	0.600/0.900	0.827/0.988
	0.3	0.495/0.878	0.371/0.794	0.508/0.879	0.787/0.981
	90	0.630/0.909	0.494/0.840	0.643/0.909	0.882/0.986
JC	80	0.597/0.904	0.462/0.832	0.613/0.904	0.861/0.982
	70	0.559/0.899	0.427/0.825	0.578/0.900	0.842/0.980
MB+JC	5×5,80	0.519/0.872	0.394/0.788	0.535/0.874	0.822/0.972
GN+JC	10,80	0.384/0.544	0.277/0.443	0.397/0.579	0.765/0.804
RS+JC	0.5,80	0.341/0.850	0.242/0.757	0.361/0.854	0.702/0.959
MB+GN+JC	5×5,10,80	0.322/0.520	0.232/0.418	0.335/0.555	0.729/0.790
MB+RS+JC	5×5,0.5,80	0.321/0.828	0.228/0.732	0.345/0.835	0.694/0.946
GN+RS+JC	10,0.5,80	0.247/0.645	0.170/0.534	0.269/0.674	0.661/0.852
	5×5,10,0.5,90	0.256/0.596	0.177/0.487	0.272/0.629	0.669/0.829
MB+GN+RS+JC	5×5,10,0.5,80	0.217/0.618	0.150/0.506	0.236/0.650	0.649/0.840
	5×5,10,0.5,70	0.201/0.621	0.140/0.509	0.217/0.653	0.632/0.842