Sigmoid函数的低复杂度概率分段线性拟合法

doi:10.19665/j.issn1001-2400.2020.03.008

摘要/Abstract

摘要：

针对传统的Sigmoid激活函数拟合方法准确度不高、消耗大量资源等问题,提出了一种基于各层神经元值分布概率的Sigmoid函数分段线性拟合方法,以便在仅使用加法电路的情况下,提高神经网络的识别精度。首先以Sigmoid函数的二阶导数为基础,将Sigmoid函数划分为3个固定区域;其次根据每层神经元值的概率变化,将每个固定区域的曲线再划分成不同数量、长度的子区域,以减少近似误差, 提高识别精度。分段线性函数斜率设为2 ^- ⁿ,有效地降低了Sigmoid函数的硬件实现复杂度。最后,设计了拟合函数的硬件电路结构,并在Xilinx FPGA-XC7A200T上实现MNIST手写数字识别,对所提出的拟合方法进行验证。实验结果表明,该方法的识别准确率在深度神经网络中约达到97.45%,而在卷积神经网络中约达到98.42%。与其他仅使用加法电路的拟合方法对比,准确率分别提高了约0.84%和0.57%。

关键词: Sigmoid函数, 概率, 神经网络, 分段线性拟合, 现场可编程门阵列

Abstract:

In order to improve the network recognition accuracy in the low complexity condition, a piecewise linear sigmoid function approximation based on the distribution probability of the neurons’ values is proposed only with one addition circuit. The sigmoid function is first divided into three fixed regions. Second, according to the neurons’ values distribution probability, the curve in each region is segmented into sub-regions to reduce the approximation error and improve the recognition accuracy. The slope of the piecewise linear function is set as 2-n, effectively reducing the hardware implementation complexity. Experiments performed on Xilinx’s FPGA-XC7A200T implement the MNIST handwritten digits recognition. The results show that the proposed method achieves a 97.45% recognition accuracy in a deep neural network and 98.42% in a convolutional neural network, up to 0.84% and 0.57% higher than other approximation methods only with one addition circuit.

Key words: Sigmoid function, probability, neural networks, piecewise linear approximation, field programmable gate array

中图分类号:

TN402

NGUYEN Van-Truong,蔡觉平,魏琳育,褚洁. Sigmoid函数的低复杂度概率分段线性拟合法[J]. 西安电子科技大学学报, 2020, 47(3): 58-65.

NGUYEN Van-Truong,CAI Jueping,WEI Linyu,CHU Jie. Low complexity probability-based piecewise linear approximation of the sigmoid function[J]. Journal of Xidian University, 2020, 47(3): 58-65.

图/表 13

图1

图2

表1

图3

表2

图4

表3

表4

表5

表6

表7

表8

图5

参考文献 16

[1]	KANG M, LIM S, GONUGONDLA S, et al. An In-memory VLSI Architecture for Convolutional Neural Networks[J]. IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 2018,8(3):494-505. doi: 10.1109/JETCAS.2018.2829522
[2]	LIU Q, LIU J, SANG R., et al. Fast Neural Network Training on FPGA Using Quasi-Newton Optimization Method[J]. IEEE Transactions on Very Large Scale Integration (VLSI) Systems, 2018,26(8):1575-1579. doi: 10.1109/TVLSI.2018.2820016
[3]	HAJDUK Z. Hardware Implementation of Hyperbolic Tangent and Sigmoid Activation Function[J]. Bulletin of the Polish Academy of Sciences: Technical Sciences, 2018,66(5):563-577.
[4]	DEL CAMPO I, FINKER R, ECHANOBE J, et al. Controlled Accuracy Approximation of Sigmoid Function for Efficient FPGA-based Implementation of Artificial Neurons[J]. Electronics Letters, 2013,49(25):1598-1600. doi: 10.1049/el.2013.3230
[5]	NAMIN A H, LEBOEUF K, MUSCEDERE R, et al. Efficient Hardware Implementation of the Hyperbolic Tangent Sigmoid Function [C]//Proceedings of the IEEE International Symposium on Circuits and Systems. Piscataway: IEEE, 2009: 2117-2120.
[6]	TIWARI V, KHARE N. Hardware Implementation of Neural Network with Sigmoidal Activation Functions Using CORDIC[J]. Microprocessors and Microsystems, 2015,39(6):373-381. doi: 10.1016/j.micpro.2015.05.012
[7]	NASCIMENTO I, JARDIM R, MORGADO D F. A New Solution to the Hyperbolic Tangent Implementation in Hardware: Polynomial Modeling of the Fractional Exponential Part[J]. Neural Computing and Applications, 2013,23(2):363-369. doi: 10.1007/s00521-012-0919-0
[8]	ARMATO A, FANUCCI L, SCILINGO E P, et al. Low-error Digital Hardware Implementation of Artificial Neuron Activation Functions and Their Derivative[J]. Microprocessors and Microsystems, 2011,35(6):557-567. doi: 10.1016/j.micpro.2011.05.007
[9]	BAJGER M, OMONDI A. Low-error, High-speed Approximation of the Sigmoid Function for Large FPGA Implementations[J]. Journal of Signal Processing Systems, 2008,52(2):137-151. doi: 10.1007/s11265-007-0140-z
[10]	SAVICH A W, MOUSSA M, AREIBI S. The Impact of Arithmetic Representation on Implementing MLP-BP on FPGAs: a Study[J]. IEEE Transactions on Neural Networks, 2007,18(1):240-252. doi: 10.1109/TNN.2006.883002
[11]	GOMAR S, MIRHASSANI M, AHMADI M. Precise Digital Implementations of Hyperbolic Tanh and Sigmoid Function [C]//Proceedings of the Asilomar Conference on Signals, Systems and Computers. Washington: IEEE Computer Society, 2016: 1586-1589.
[12]	NGAH S, BAKAR R A. Sigmoid Function Implementation Using the Unequal Segmentation of Differential Lookup Table and Second Order Nonlinear Function[J]. Journal of Telecommunication, Electronic and Computer Engineering (JTEC), 2017,9(2):103-108.
[13]	MITRA S, CHATTOPADHYAY P. Challenges in Implementation of ANN in Embedded System [C]//Proceedings of the International Conference on Electrical, Electronics, and Optimization Techniques. Piscataway: IEEE, 2016: 1794-1798.
[14]	ZAMANLOOY B, MIRHASSANI M. An Analog CVNS-based Sigmoid Neuron for Precise Neurochips[J]. IEEE Transactions on Very Large Scale Integration (VLSI) Systems, 2017,25(3):894-906. doi: 10.1109/TVLSI.2016.2615306
[15]	DATTA D, AGARWAL S, KUMAR V, et al. Design of Current Mode Sigmoid Function and Hyperbolic Tangent Function [C]//Proceedings of the Communications in Computer and Information Science. Heidelberg: Springer, 2019: 47-60.
[16]	LECUN Y. MNIST Handwritten Digit Database[DB/OL]. [ 2019-10-12]. http://yann. lecun. com/exdb/mnist/.

区Ⅰ的概率在从0% 到30%之间			区Ⅰ的概率在从30% 到 70%之间			区Ⅰ的概率在从70%到 100%之间
区间	n	b_i	区间	n	b_i	区间	n	b_i
[0, 1.1)	2	0.4877	[0, 1)	2	0.4906	[0, 0.9)	2	0.4931
[1.1, 2.2)	3	0.6231	[1, 1.5)	3	0.6182	[0.9, 1.3)	3	0.6110
[2.2, 2.6)	4	0.7657	[1.5, 2.2)	3	0.6293	[1.3, 1.7)	3	0.6283
[2.6, 3.2)	4	0.7647	[2.2, 2.6)	4	0.7657	[1.7, 2.2)	3	0.6293
[3.2, 4.1)	5	0.8586	[2.6, 3.2)	4	0.7647	[2.2, 3.6)	4	0.7583
[4.1, 5)	7	0.9531	[3.2, 5)	6	0.9153	[3.6, 5)	6	0.9175

资源类型	已使用	总资源	占用比例/%
Slice LUT	145	134600	0.110
Slice Register	44	269200	0.001
Block RAM	0	365	0.000
DSP	0	740	0.000

拟合函数	段数		最大绝对误差		平均绝对误差
拟合函数	区Ⅰ	区Ⅱ	区Ⅰ	区Ⅱ	区Ⅰ	区Ⅱ
区Ⅰ的概率小于30%	4	8	0.0125	0.0039	0.00755	0.00163
区Ⅰ的概率在30% 到 70% 之间	6	6	0.0121	0.0045	0.00519	0.00249
区Ⅰ的概率大于70%	8	4	0.0120	0.0101	0.00418	0.00396

拟合方法	区间	总段数	绝对误差
拟合方法	区间	总段数	最大绝对误差	平均绝对误差
Armato [8]	[-8, 8]	16	0.00788	0.00107
Ngah [12]	[-4, 4]		0.02200	0.00770
Gomar [11]	[-4, 4]		0.00870	0.00580
Mitra [13]	[-9.35, 9.35]	14	0.01270	0.00150
Zamanlooy[14]	[-8, 8]	6	0.01890	0.00590
Campo[4]	[-4.59, 4.59]	12	0.02800	0.00430
Savich [10]	[-8, 8]	5	0.06790	0.02630
提出的方法	[-5, 5]	12	0.01250	0.00420

网络层类型	神经元数量	神经元值的分布概率/%
网络层类型	神经元数量	区Ⅰ	区Ⅱ	区Ⅲ
Input	784
Hidden1	576	54.69	43.93	1.38
Hidden2	450	98.96	1.04	0.00
Hidden3	300	92.63	7.37	0.00
Hidden4	120	42.25	57.50	0.25
Hidden5	80	26.29	71.45	2.26
Output	10	0.80	11.00	88.20