The invention discloses a method for realizing a
web crawler based on a large front-end technology, which comprises the following steps of: firstly, starting a Puppeteer by using a NodeJS to carry out a web
crawling service which can simulate the operation of real human beings, click any position of a page, roll the page, drag elements, read page contents, intercept a webpage, intercept an HTTP (
Hyper Text Transport Protocol) request and the like, namely, what you see is what you get; on the basis of the characteristic, the development speed which is incomparable to other crawlers can be realized; secondly, user-oriented
visual interaction service is carried out, so that the user is helped to control a Puppeteer through the application, and webpage content capturing, webpage screenshot, network request interception and the like are carried out; the data storage service is used for storing related information of a user, so that the user can conveniently check own operation history, quickly initialize an application, quickly switch a crawler context environment and the like, and the optimal user experience is achieved; and finally, the pictures can be analyzed and processed through a TensorFlow
deep learning framework, such as
character recognition (used for
verification code
verification), image classification (providing personalized capture strategies for different users) and the like.