Abstract:

In this paper web scraping technologies are discussed.The advantages of Web data collection technology for high speed and accuracy conversion of unstructured data into structured data are pointed out.The principles of the web scraping at HTTP level are introduced with emphasis on the technical solutions to Python-based web scraping.Web scraping system consists of two modules:HTTP interaction module and data analysis module.

Key words: Web scraping;screen scraping;HTTP;Python;regex;XPath