The application discloses a method and
system for generating intelligent crawlers based on a large
language model and an MCP protocol, and belongs to the technical field of
network data collection. The method solves the problem that the prior art cannot effectively integrate an LLM and a browser
interaction protocol, resulting in the inability to fully exert the capability of the LLM in the dynamic webpage analysis and anti-
crawling strategy generation links. The method comprises the following steps: analyzing the collection requirement based on a large
language model and generating a standardized requirement description document; realizing the interaction between the large
language model and the browser based on an MCP protocol; and performing complete DOM
tree structure analysis and
processing on a page through a crawler script generation
system, and performing quality checking and intelligent repairing on the generated crawler script. In the application, the complete DOM tree and
dynamic data after browser rendering are obtained through the MCP protocol, and the large language model can be called to automatically analyze element positioning strategies, so that the collection script can be adaptively generated, and the efficiency, intelligence and
automation of webpage data collection are ensured.