基于Spark框架的CFSFDP改进算法

doi:10.16180/j.cnki.issn1007-7820.2019.05.008

Abstract

Abstract:

CFSFDP algorithm based on density is a clustering algorithm. In order to rid dependency on artificial selection of decision graph, this paper used the idea of slope to calculate the demarcation point of clustering center points and un-clustering center points. This improvement eliminated personal equation and realized auto-calculation of center points. Parallel processing for the algorithm was conducted through the Spark framework. The experiments showed that this algorithm was applicable to clustering analysis of mass data, since it improved efficiency by eliminating personal equation and displayed great speed up ratio and extendibility after paralleling.

Key words: Spark, CFSFDP algorithm, decision diagram, density peaks, clustering, parallel

CLC Number:

TP301.6

LI Qi,ZHANG Xin,ZHANG Pingkang,ZHANG Hang. Improved CFSFDP Algorithm Based on Spark Framework[J].Electronic Science and Technology, 2019, 32(5): 38-44.

Figures/Tables 12

Figure 1.

Figure 2.

Figure 3.

Figure 4.

Figure 5.

Figure 6.

Table 1

Figure 7.

Table 2

Figure 8.

Figure 9.

Figure 10.

References 16

[1]	孙吉贵, 刘杰, 赵连宇 . 聚类算法研究[J]. 软件学报, 2008,19(1):48-61.
	Sun Jigui, Liu Jie, Zhao Lianyu . Clustering algorithms research[J]. Journal of Software, 2008,19(1):48-61.
[2]	Jain A K, Dubes R C . Algorithms for clustering data[M]. New Jersy:Prentice Hall, 1988.
[3]	Zaharia M, Chowdhury M, Franklin M J, et al. Spark: cluster computing with working sets[C]. USENIX Association:Usenix Conference on Hot Topics in Cloud Computing, 2010.
[4]	Dean J, Ghemawat S . MapReduce: simplified data processing on large clusters[J]. Communications of the ACM, 2008,51(1):10-17.
[5]	Engle C, Lupher A, Xin R, et al. Shark:fast data analysis using coarse-grained distributed memory[C]. Scottsdale:ACM SIGMOD International Conference on Management of Data,ACM, 2012.
[6]	Rodriguez A, Laio A . Clustering by fast search and find of density peaks[J]. Science, 2014,344(6191):1492-1503.
[7]	杨洁, 王国胤, 庞紫玲 . 密度峰值聚类相关问题的研究[J]. 南京大学学报, 2017,53(4):791-801.
	Yang Jie, Wang Guoyin, Pang Ziling . Relative researches of clustering by fast search and find of density peaks[J]. Journal of Nanjing University, 2017,53(4):791-801.
[8]	蒋礼青, 张明新, 郑金龙 , 等. 快速搜索与发现密度峰值聚类算法的优化研究[J]. 计算机应用研究, 2016,33(11):3251-3254.
	Jiang Liqing, Zhang Mingxin, Zheng Jinlong , et al. Optimization of clustering by fast search and find of density peaks[J]. Application Research of Computers, 2016,33(11):3251-3254.
[9]	李敏, 张桂珠 . 密度峰值优化初始中心的K-means算法[J]. 计算机应用与软件, 2017,34(3):212-217.
	Li Min, Zhang Guizhu . K-means algorithm of optimized initial center by density peaks[J]. Computer Applications and Software, 2017,34(3):212-217.
[10]	Wang S, Wang D, Li C , et al. Clustering by fast search and find of density peaks with data field[J]. Chinese Journal of Electronics, 2016,25(3):397-402. doi: 10.1049/cje.2016.05.001
[11]	孙昊, 张明新, 戴娇 , 等. 基于网格的快速搜寻密度峰值的聚类算法优化研究[J]. 计算机工程与科学, 2017,39(5):964-970.
	Sun Hao, Zhang Minxing, Dai Jiao , et al. Optimization of grid based clustering by fast search and find of density peaks[J]. Computer Engineering & Science, 2017,39(5):964-970.
[12]	戴娇, 张明新, 郑金龙 , 等. 基于密度峰值的快速聚类算法优化[J]. 计算机工程与设计, 2016,37(11):2979-2984.
	Dai Jiao, Zhang Minxing, Zheng Jinlong , et al. Optimization of fast clustering algorithm based on density peaks[J]. Computer Engineering and Design, 2016,37(11):2979-2984.
[13]	张天宇, 谌志群, 黄孝喜 , 等. 基于改进CFSFDP算法的电信投诉文本聚类方法[J]. 电子科技, 2017,30(10):93-96.
	Zhang Tianyu, Chen Zhiqun, Huang Xiaoxi , et al. Clustering method for telecom complaints text based on improved CFSFDP algorithm[J]. Electronic Science and Technology, 2017,30(10):93-96.
[14]	张文开 . 基于密度的层次聚类算法研究[D]. 合肥:中国科学技术大学, 2015.
	Zhang Wenkai . Research on density-based hierarchical clustering algorithm[D]. Hefei:University of Science and Technology of China, 2015.
[15]	马春来, 单洪, 马涛 , 等. 一种基于CFSFDP改进算法的重要地点识别方法研究[J]. 计算机应用研究, 2017, 34(1):136-140.
	Ma Chunlai, Shan Hong, Ma Tao , et al. Research on important places identification method based on improved CFSFDP algorithm[J]. Application Research of Computers, 2017,34(1):136-140.
[16]	Rong C . Using mahout for clustering wikipedia's latest articles:a comparison between K-means and fuzzy C-means in the cloud[C]. Athens:IEEE Third International Conference on Cloud Computing Technology and Science, IEEE Computer Society, 2011.

数据集	样本数	类别数	维度
Iris	150	3	4
Wine	178	3	13
R15	600	15	2
Aggregation	788	7	2

数据集	B_n值	分界点
Iris	0.57	3
Wine	0.76	3
R15	3.55	15
Aggregation	1.21	7

Improved CFSFDP Algorithm Based on Spark Framework

RichHTML

PDF (PC)

Like

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 12

References 16

Related Articles 15

Metrics

Comments

Recommended 10

[1]	JIN Xiao,WU Fei,YAN Song,LU Wenxia,ZHANG Zhongyi. Fingerprint Location Method of Metro Station Based on GAWK-means [J]. Electronic Science and Technology, 2022, 35(2): 34-39.
[2]	GENG Zhaoqian,ZHU Huming,LI Xuming,CHEN Meiqing,YANG Guipeng. A Review: Radar Signal Processing Based on High Performance Computing [J]. Electronic Science and Technology, 2021, 34(9): 1-6.
[3]	XU Hangfan,LIU Cong,TANG Jiangang,PENG Dunlu. Accelerated Spectral Clustering Based on Improved Landmark Selection [J]. Electronic Science and Technology, 2021, 34(5): 47-53.
[4]	WANG Yang,WANG Xiaolei,YUAN Ziang,YUAN Ruming. A Matrix Multiplication Mapping Technology Based on NOC Multi-Core System [J]. Electronic Science and Technology, 2021, 34(5): 54-60.
[5]	GONG Jialiang,TANG Qingshan,BAI Chuang. Design of Efficiency Parallel HDMI 2.0 Encoder Based on FPGA [J]. Electronic Science and Technology, 2021, 34(4): 34-40.
[6]	HE Zhonglin,ZHOU Ping. Research on Sorting of Retired Lithium Battery Based on Radial Basis Function Neural Network [J]. Electronic Science and Technology, 2021, 34(2): 38-44.
[7]	HU Qiongqiong,SAN Hongjun,CHEN Jiupeng,XIE Feiya,CHEN Zhongping,LI Pengyu. Motions Analysis and Simulation of a Parallel Robot with Four Degrees of Freedom [J]. Electronic Science and Technology, 2021, 34(11): 46-54.
[8]	SHANGGUAN Xuanfeng,LIU Yongjian,YANG Tingyu,WEI Jinsong. Analysis of Operation Characteristics and Commutating Performance of Four-Poles Single-Phase Universal Motor [J]. Electronic Science and Technology, 2021, 34(10): 1-7.
[9]	LI Lin,XIE Ming,HUANG Song. Design of Modular Parallel Current Sharing Single-Phase Inverter [J]. Electronic Science and Technology, 2021, 34(10): 27-31.
[10]	CHENG Huaxiang,CHEN Jie,ZHANG Chi. Optimal Design of 6-bar Parallel Machine Tool Based on HyperWorks [J]. Electronic Science and Technology, 2020, 33(9): 25-30.
[11]	LU Huan. Research on Improving the Efficiency of Parallel Charger Based on Constraint Optimizations [J]. Electronic Science and Technology, 2020, 33(8): 53-58.
[12]	YANG Shanshan,ZHANG Daxing,GUO Jiawei,WANG Shitiao. Copy-move Detection Algorithm Combining Keypoints and Block Advantages [J]. Electronic Science and Technology, 2020, 33(3): 38-43.
[13]	MIAO Ran,LI Feifei,CHEN Qiu. Scene Recognition Algorithm Based on Convolutional Neural Networks and Multi-Scale space Encoding [J]. Electronic Science and Technology, 2020, 33(12): 54-58.
[14]	SUN Zhen,WANG Kai,WANG Yagang. A Modified Routing Algorithm Based on Time Competition Clustering [J]. Electronic Science and Technology, 2019, 32(8): 27-32.
[15]	ZHANG Changqing,YANG Nan. Design of Real-time Road Condition Detection System Based on Big Data Analysis of Vehicle Network [J]. Electronic Science and Technology, 2019, 32(8): 66-70.