Abstract:

A design and implementation scheme of lexical analyzer based on language dictionary is proposed for lexical analysis in program source code similarity measurements.The program source codes are converted first into a string,which is decomposed character by character to obtain each token of the program source codes,whose type is judged for lexical analysis.Experimental results show the scheme can effectively analyze the lexical structure of the program source codes.

Key words: lexical analyzer;Token;language dictionary;source code analyzing