The application discloses a kind of based on
dynamic feature identification intelligent crawler safety collection method and
system, including URL manager, dispatcher,
resolver, downloader and data storage.Will be added and maintained to be crawled URL set and the URL set of having been crawled by URL manager, realize URL deduplication;Dispatcher is used for coordinating and allocating download task;
Resolver is used for analyzing the page type of target website, structure,
request response feature, automatically adjusting
crawling parameter, optimizing page
access frequency, automatic
verification Captcha, analyzing page depth, compliance
verification crawling content;Downloader is used for simulating user behavior downloading page;Data storage is used for storing data.The application has the advantages that: using
dynamic feature identification method completes the dynamic adjustment of
crawling parameter,
access frequency self-
adaptation,
Captcha automatic processing, crawling process personification, crawling page depth controllable, crawling content compliance, realizes the adaptability, high efficiency and safety of
network data acquisition process.