Dynamic URL Collection via Script Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for collecting URLs from web pages, known as static collection, often omit invoked URLs and include invalid ones, as they rely solely on parsing the source code without executing the scripts.
Innovation Solution
A method and apparatus that dynamically collect URLs by executing the source code of a web page, including resource URLs invoked during rendering, event tags, and script-executed URLs, while also parsing exposed URLs to ensure comprehensive collection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If URL collection is performed by parsing source code only (static collection), then collection process is simple and fast, but invoked URLs are omitted and invalid URLs are included
Solution Approach 1:
The patent transitions from static URL collection (parsing source code) to dynamic URL collection (executing scripts). The system executes JavaScript codes and other scripts embedded in web pages to dynamically retrieve URLs, ensuring that invoked URLs are captured while filtering out invalid ones through validation mechanisms.
Solution Approach 2:
The patent replaces the mechanical parsing approach with a dynamic execution approach. Instead of merely reading and parsing source code, the system executes the scripts and codes embedded in web pages to dynamically retrieve URLs, thereby capturing URLs that would otherwise be omitted through static analysis.
2Reliability
If URL collection is performed by executing source codes (dynamic collection), then URL completeness improves, but collection complexity and time increase
Solution Approach 1:
The patent segments the URL collection process into multiple independent modules: static parsing module, dynamic execution module, validation module, and deduplication module. Each module handles specific aspects of URL collection, making the overall complex process manageable and maintainable while ensuring comprehensive URL retrieval.
Solution Approach 2:
The patent performs preliminary actions by executing scripts and codes before URL extraction. The system pre-processes the web page by running embedded scripts to generate dynamic content, ensuring that all URLs are revealed before the actual collection process begins, thereby improving completeness.
3Measurement precision
If all source codes are executed to collect URLs, then URL accuracy improves, but processing time increases
Solution Approach 1:
The patent applies partial execution by selectively executing only the necessary portions of source codes related to URL retrieval. The system identifies and executes only the scripts and codes that are likely to contain URL information, rather than executing every possible code, thereby balancing precision with time efficiency.
Solution Approach 2:
The patent implements feedback mechanisms where the system validates collected URLs and uses this feedback to refine the execution process. Invalid URLs are identified and filtered out, and the system learns from these results to optimize future execution strategies, reducing unnecessary processing time while maintaining high precision.
Data Source
AI summary
A URL collecting method includes accessing, by a URL collecting apparatus, a web server of a first URL; receiving, by the URL collecting apparatus, a first web page from the web server; and a URL dynamic collecting step of collecting, by the URL collecting apparatus, one or more URLs invoked while performing some or all of source codes of the first web page.