基于Spark的改进随机森林算法

doi:10.16180/j.cnki.issn1007-7820.2019.04.013

Abstract

Abstract:

For the classical random forest algorithm based on single machine couldn't meet the demand of dealing with massive data, an improved random forest algorithm based on Spark was designed and implemented by using Spark distributed memory computing technology. Firstly, after calculating the importance of features the features were divided into public features, unique features, and non-important features;. Then, randomly features were selected in each feature subspace in order and proportion. Finally, the experiment was performed through Spark clusters to analyze the improved classification performance, speedup ratio and efficiency of the random forest algorithm. The result demonstrated that the improved algorithm could improve the efficiency of random forest construction and could be used to solve the massive data mining problem with good scalability.

Key words: random forest, spark, feature space, ReliefF algorithm, high dimensional data, classification model

CLC Number:

TP311.13

SUN Yue,YUAN Jian. Improved Random Forest Algorithm Based on Spark[J].Electronic Science and Technology, 2019, 32(4): 60-64.

Figures/Tables 3

References 16

[1]	Kulkarni V Y, Sinha P K . Efficient learning of random forest classifier using disjoint partitioning approach[J]. Proceedings of the World Congress on Engineering, 2013,2(5):1-5.
[2]	Mi Y . Imbalanced classification based on active learning SMOTE[J]. Research Journal of Applied Sciences Engineering & Technology, 2013,5(3):944-949.
[3]	Amaratunga D Cabrera J Lee Y S . Enriched random forests[J]. Bioinformatics, 2008,24(18):2010-2014. doi: 10.1093/bioinformatics/btn356
[4]	Xu B X, Huang J Z, Williams G , et al. Classifying very high-dimensional data with random forests built from small subspaces[J]. International Journal of Data Warehousing and Mining, 2011,8(2):44-63. doi: 10.4018/jdwm.2012040103
[5]	Ye Y M, Wu Q Y, Huang J Z , et al. Stratified sampling for feature subspace selection in random forests for high dimensional data[J]. Pattern Recognition, 2013,46(3):769-787. doi: 10.1016/j.patcog.2012.09.005
[6]	Sun K, Miao W, Zhang X, et al. An improvement to feature selection of random forests on Spark [C].Chengdu:2014 IEEE 17 ^th International Conference on Computational Science and Engineering , 2014.
[7]	Wu X, Zhu X, Wu G Q . Data mining with big data[J]. IEEE Transactions on Knowledge Data Engineering, 2014,26(1):97-107. doi: 10.1109/TKDE.2013.109
[8]	Kuang L, Hao F, Yang L T , et.al. A tensor-based approach for big data representation and dimensionality reduction[J]. IEEE Transactions on Emerge Topics Computer, 2014,2(3):280-291. doi: 10.1109/TETC.2014.2330516
[9]	Zhang C, Yuan D. Fast fine-grained air quality index level prediction using random forest algorithm on cluster computing of Spark [C].Beijing: IEEE,UIC-ATC-ScalCom-CBDCom-Iop, 2015.
[10]	Dean J, Ghemawat S . MapReduce: simplified data processing on large clusters[J]. Communications of the ACM, 2008,51(1):107-113.
[11]	夏卫雷, 王立松 . 基于MapReduce的并行蚁群算法研究与实现[J]. 电子科技, 2013,26(2):146-149. doi: 10.3969/j.issn.1007-7820.2013.02.046
	Xia Weilei, Wang Lisong . Research on and implementation of parallel ant colony algorithm based on MapReduce[J]. Electronic Science and Technology, 2013,26(2):146-149. doi: 10.3969/j.issn.1007-7820.2013.02.046
[12]	Dimple B, Sudarshan T . IBM text analytics on Apache Spark[M]. San Francisco:Saprk Summit, 2014.
[13]	Li W, Cheng H L, Peng Y. Visualized data mining platform based on the Spark [C].Hangzhou:Proceedings of the 16 ^th System Simulation Technology and Application , 2014.
[14]	Bian H Q, Chen Y G, Du X Y . Equal-join optimization on Spark[J]. Journal of East China Normal University :Natural Science, 2014,5(1):263-270.
[15]	Zhang J, Li T, Da R . A parallel method for computing rough set approximations[J]. Information Sciences, 2012,194(5):209-223. doi: 10.1016/j.ins.2011.12.036
[16]	Zhu Weisheng, Wang Peng . Large-scale image retrieval solution based on hadoop cloud computing platform[J]. Journal of Computer Aplliction, 2014,34(3):695-699.

	平均精度/%	平均CPU耗时/ms
RF	86.885	42.159
IRFA	86.282	27.897

Improved Random Forest Algorithm Based on Spark

RichHTML

PDF (PC)

Like

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 3

References 16

Related Articles 8

Metrics

Comments

Recommended 10

[1]	SHANGGUAN Xuanfeng,LIU Yongjian,YANG Tingyu,WEI Jinsong. Analysis of Operation Characteristics and Commutating Performance of Four-Poles Single-Phase Universal Motor [J]. Electronic Science and Technology, 2021, 34(10): 1-7.
[2]	JIN Xin,FENG Yi,YOU Xuexi,WANG Jiaxin. Research on Information Security Equipment Deployment Guarantee Technology [J]. Electronic Science and Technology, 2020, 33(8): 80-86.
[3]	LIU Yanwen,WEI Yun. Research of Emotional Analysis Based on LDA Topic Model [J]. Electronic Science and Technology, 2020, 33(7): 12-16.
[4]	SUN Liping,ZHANG Ximeng,HE Rui,LI Jiaqi. Near-infrared Scanning Polysaccharide Content Classification of Auricularia Auricular Based on SVM [J]. Electronic Science and Technology, 2019, 32(8): 16-21.
[5]	LAN Jiman. Design of Data Mining System Based on Cloud Computing [J]. Electronic Science and Technology, 2019, 32(8): 70-74.
[6]	LI Yuan. Research on Distributed Mobile Phone Signaling Data Acquisition and Analysis Technology [J]. Electronic Science and Technology, 2019, 32(6): 78-81.
[7]	LI Qi,ZHANG Xin,ZHANG Pingkang,ZHANG Hang. Improved CFSFDP Algorithm Based on Spark Framework [J]. Electronic Science and Technology, 2019, 32(5): 38-44.
[8]	ZHAO Man. Intrinsically Safe Behavior Assessment of Explosion-proof Electrical Apparatus and Electric Circuit [J]. , 2013, 26(5): 174-.