Sigmoid函数的低复杂度概率分段线性拟合法

doi:10.19665/j.issn1001-2400.2020.03.008

Abstract

Abstract:

In order to improve the network recognition accuracy in the low complexity condition, a piecewise linear sigmoid function approximation based on the distribution probability of the neurons’ values is proposed only with one addition circuit. The sigmoid function is first divided into three fixed regions. Second, according to the neurons’ values distribution probability, the curve in each region is segmented into sub-regions to reduce the approximation error and improve the recognition accuracy. The slope of the piecewise linear function is set as 2-n, effectively reducing the hardware implementation complexity. Experiments performed on Xilinx’s FPGA-XC7A200T implement the MNIST handwritten digits recognition. The results show that the proposed method achieves a 97.45% recognition accuracy in a deep neural network and 98.42% in a convolutional neural network, up to 0.84% and 0.57% higher than other approximation methods only with one addition circuit.

Key words: Sigmoid function, probability, neural networks, piecewise linear approximation, field programmable gate array

CLC Number:

TN402

NGUYEN Van-Truong,CAI Jueping,WEI Linyu,CHU Jie. Low complexity probability-based piecewise linear approximation of the sigmoid function[J].Journal of Xidian University, 2020, 47(3): 58-65.

Figures/Tables 13

References 16

[1]	KANG M, LIM S, GONUGONDLA S, et al. An In-memory VLSI Architecture for Convolutional Neural Networks[J]. IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 2018,8(3):494-505. doi: 10.1109/JETCAS.2018.2829522
[2]	LIU Q, LIU J, SANG R., et al. Fast Neural Network Training on FPGA Using Quasi-Newton Optimization Method[J]. IEEE Transactions on Very Large Scale Integration (VLSI) Systems, 2018,26(8):1575-1579. doi: 10.1109/TVLSI.2018.2820016
[3]	HAJDUK Z. Hardware Implementation of Hyperbolic Tangent and Sigmoid Activation Function[J]. Bulletin of the Polish Academy of Sciences: Technical Sciences, 2018,66(5):563-577.
[4]	DEL CAMPO I, FINKER R, ECHANOBE J, et al. Controlled Accuracy Approximation of Sigmoid Function for Efficient FPGA-based Implementation of Artificial Neurons[J]. Electronics Letters, 2013,49(25):1598-1600. doi: 10.1049/el.2013.3230
[5]	NAMIN A H, LEBOEUF K, MUSCEDERE R, et al. Efficient Hardware Implementation of the Hyperbolic Tangent Sigmoid Function [C]//Proceedings of the IEEE International Symposium on Circuits and Systems. Piscataway: IEEE, 2009: 2117-2120.
[6]	TIWARI V, KHARE N. Hardware Implementation of Neural Network with Sigmoidal Activation Functions Using CORDIC[J]. Microprocessors and Microsystems, 2015,39(6):373-381. doi: 10.1016/j.micpro.2015.05.012
[7]	NASCIMENTO I, JARDIM R, MORGADO D F. A New Solution to the Hyperbolic Tangent Implementation in Hardware: Polynomial Modeling of the Fractional Exponential Part[J]. Neural Computing and Applications, 2013,23(2):363-369. doi: 10.1007/s00521-012-0919-0
[8]	ARMATO A, FANUCCI L, SCILINGO E P, et al. Low-error Digital Hardware Implementation of Artificial Neuron Activation Functions and Their Derivative[J]. Microprocessors and Microsystems, 2011,35(6):557-567. doi: 10.1016/j.micpro.2011.05.007
[9]	BAJGER M, OMONDI A. Low-error, High-speed Approximation of the Sigmoid Function for Large FPGA Implementations[J]. Journal of Signal Processing Systems, 2008,52(2):137-151. doi: 10.1007/s11265-007-0140-z
[10]	SAVICH A W, MOUSSA M, AREIBI S. The Impact of Arithmetic Representation on Implementing MLP-BP on FPGAs: a Study[J]. IEEE Transactions on Neural Networks, 2007,18(1):240-252. doi: 10.1109/TNN.2006.883002
[11]	GOMAR S, MIRHASSANI M, AHMADI M. Precise Digital Implementations of Hyperbolic Tanh and Sigmoid Function [C]//Proceedings of the Asilomar Conference on Signals, Systems and Computers. Washington: IEEE Computer Society, 2016: 1586-1589.
[12]	NGAH S, BAKAR R A. Sigmoid Function Implementation Using the Unequal Segmentation of Differential Lookup Table and Second Order Nonlinear Function[J]. Journal of Telecommunication, Electronic and Computer Engineering (JTEC), 2017,9(2):103-108.
[13]	MITRA S, CHATTOPADHYAY P. Challenges in Implementation of ANN in Embedded System [C]//Proceedings of the International Conference on Electrical, Electronics, and Optimization Techniques. Piscataway: IEEE, 2016: 1794-1798.
[14]	ZAMANLOOY B, MIRHASSANI M. An Analog CVNS-based Sigmoid Neuron for Precise Neurochips[J]. IEEE Transactions on Very Large Scale Integration (VLSI) Systems, 2017,25(3):894-906. doi: 10.1109/TVLSI.2016.2615306
[15]	DATTA D, AGARWAL S, KUMAR V, et al. Design of Current Mode Sigmoid Function and Hyperbolic Tangent Function [C]//Proceedings of the Communications in Computer and Information Science. Heidelberg: Springer, 2019: 47-60.
[16]	LECUN Y. MNIST Handwritten Digit Database[DB/OL]. [ 2019-10-12]. http://yann. lecun. com/exdb/mnist/.

区Ⅰ的概率在从0% 到30%之间			区Ⅰ的概率在从30% 到 70%之间			区Ⅰ的概率在从70%到 100%之间
区间	n	b_i	区间	n	b_i	区间	n	b_i
[0, 1.1)	2	0.4877	[0, 1)	2	0.4906	[0, 0.9)	2	0.4931
[1.1, 2.2)	3	0.6231	[1, 1.5)	3	0.6182	[0.9, 1.3)	3	0.6110
[2.2, 2.6)	4	0.7657	[1.5, 2.2)	3	0.6293	[1.3, 1.7)	3	0.6283
[2.6, 3.2)	4	0.7647	[2.2, 2.6)	4	0.7657	[1.7, 2.2)	3	0.6293
[3.2, 4.1)	5	0.8586	[2.6, 3.2)	4	0.7647	[2.2, 3.6)	4	0.7583
[4.1, 5)	7	0.9531	[3.2, 5)	6	0.9153	[3.6, 5)	6	0.9175

资源类型	已使用	总资源	占用比例/%
Slice LUT	145	134600	0.110
Slice Register	44	269200	0.001
Block RAM	0	365	0.000
DSP	0	740	0.000

拟合函数	段数		最大绝对误差		平均绝对误差
拟合函数	区Ⅰ	区Ⅱ	区Ⅰ	区Ⅱ	区Ⅰ	区Ⅱ
区Ⅰ的概率小于30%	4	8	0.0125	0.0039	0.00755	0.00163
区Ⅰ的概率在30% 到 70% 之间	6	6	0.0121	0.0045	0.00519	0.00249
区Ⅰ的概率大于70%	8	4	0.0120	0.0101	0.00418	0.00396

拟合方法	区间	总段数	绝对误差
拟合方法	区间	总段数	最大绝对误差	平均绝对误差
Armato [8]	[-8, 8]	16	0.00788	0.00107
Ngah [12]	[-4, 4]		0.02200	0.00770
Gomar [11]	[-4, 4]		0.00870	0.00580
Mitra [13]	[-9.35, 9.35]	14	0.01270	0.00150
Zamanlooy[14]	[-8, 8]	6	0.01890	0.00590
Campo[4]	[-4.59, 4.59]	12	0.02800	0.00430
Savich [10]	[-8, 8]	5	0.06790	0.02630
提出的方法	[-5, 5]	12	0.01250	0.00420

网络层类型	神经元数量	神经元值的分布概率/%
网络层类型	神经元数量	区Ⅰ	区Ⅱ	区Ⅲ
Input	784
Hidden1	576	54.69	43.93	1.38
Hidden2	450	98.96	1.04	0.00
Hidden3	300	92.63	7.37	0.00
Hidden4	120	42.25	57.50	0.25
Hidden5	80	26.29	71.45	2.26
Output	10	0.80	11.00	88.20

Low complexity probability-based piecewise linear approximation of the sigmoid function

RichHTML

PDF (PC)

Like

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 13

References 16

Related Articles 15

Metrics

Comments

Recommended 10

网络层类型	大小	神经元值的分布概率/%
网络层类型	大小	区Ⅰ	区ⅡI	区Ⅲ
Input	1×32×32
Conv_1	6×28×28	63.15	32.23	4.62
Pool_1	6×14×14
Conv_2	12×10×10	26.46	58.27	15.27
Pool_2	12×5×5
Full_con_1	300
Full_con_2	120	53.09	46.77	0.14
Output	10	2.20	10.72	87.08

拟合方法	乘法	加法	除法
拟合方法	DNN/CNN	DNN/CNN	DNN/CNN
Armato [8]	1536/6034	1536/6034	0/0
Ngah [12]	3072/12068	3072/12068	0/0
Gomar [11]	0/0	3072/12068	1536/6034
Campo[4]	1536/6034	1536/6034	0/0
Savich [10]	0/0	1536/6034	0/0
提出的方法	0	1536/6034	0

[1]	LV Wenkai,YANG Pengfei,DING Yunqing,ZHANG Heyu,ZHENG Tianyang. JEDERL:A task scheduling optimization algorithm for heterogeneous computing platforms [J]. Journal of Xidian University, 2021, 48(6): 67-74.
[2]	ZHANG Yuhao,CHENG Peitao,ZHANG Shuhao,WANG Xiumei. Lightweight image super-resolution with the adaptive weight learning network [J]. Journal of Xidian University, 2021, 48(5): 15-22.
[3]	SONG Jianqiao,WANG Feng,NIU Jin,SHI Zezhou,MA Junhui. Potential emotion recognition based on the fusion of the spatio-temporal neural network and facial pulse signals [J]. Journal of Xidian University, 2021, 48(4): 159-167.
[4]	HUI Haisheng,ZHANG Xueying,WU Zelin,LI Fenglian. Method for stroke lesion segmentation using the primary-auxiliary path attention compensation network [J]. Journal of Xidian University, 2021, 48(4): 200-208.
[5]	CAO Yi,CAI Xiaodong. Effective learning strategy for hard samples [J]. Journal of Xidian University, 2021, 48(3): 99-105.
[6]	SONG Chuanwang,LIU Dong,LI Enyu,ZHAO Ruishou,HAO Siyuan,SHEN Tianqi. Reactive relay selection strategy for NOMA systems [J]. Journal of Xidian University, 2021, 48(3): 56-62.
[7]	HE Shiyang,LI Hui,LI Fenghua. Optimization and implementation of the SM4 on FPGA [J]. Journal of Xidian University, 2021, 48(3): 155-162.
[8]	XIAO Yeqiu,ZHU Xinghui,ZHAO Shuangrui,REN Baoquan,SHEN Yulong. Analysis of physical layer security performance for satellite communication systems [J]. Journal of Xidian University, 2021, 48(3): 163-169.
[9]	WANG Ping,JIANG Yuze,ZHAO Guanghui. Object detection based on the multiscale location Enhancement network [J]. Journal of Xidian University, 2021, 48(3): 85-90.
[10]	GUO Zekun,TIAN Long,HAN Ning,WANG Penghui,LIU Hongwei,CHEN Bo. Radar HRRP based few-shot target recognition with CNN-SSD [J]. Journal of Xidian University, 2021, 48(2): 7-14.
[11]	LI Shan,ZHNAG Kun,SHUI Penglang. Ship length distribution modeling on China offshore and ability evaluation of radar ship classification [J]. Journal of Xidian University, 2021, 48(2): 84-91.
[12]	ZHANG Shudong,GAO Haichang,CAO Xiwen,KANG Shuai. Adaptive fast and targeted adversarial attack for speech recognition [J]. Journal of Xidian University, 2021, 48(1): 168-175.
[13]	DANG Jisheng,YANG Jun. 3D model recognition and segmentation based on multi-feature fusion [J]. Journal of Xidian University, 2020, 47(4): 149-157.
[14]	QU Bayi,LIU Yehao,ZHANG Taojing,LIU Wei,YU Dongsong,ZHOU Wei. Scheme for miniature time difference measurement with a high resolution and a large range [J]. Journal of Xidian University, 2020, 47(4): 24-30.
[15]	LI Kunlun,ZHANG Lu,XU Hongke,SONG Huansheng. Waveletdomain dilated network for fast low-dose CT image reconstruction [J]. Journal of Xidian University, 2020, 47(4): 86-93.