Web Crawler Execution Vector Pruning for Dynamic Link Discovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Crawlers face inefficiencies in retrieving dynamically generated hyperlinks due to the indiscriminate execution of computer program instructions within documents, which degrades performance and is wasteful, as many instructions do not yield hyperlinks.
Innovation Solution
A method involving static analysis to identify execution vectors within documents, executing only those program segments that lead to external entity calls and excluding others, thereby optimizing the crawling process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If indiscriminate execution of computer program instructions is performed during crawling, then dynamically generated hyperlinks are discovered, but crawler performance degrades due to unnecessary computations
Solution Approach 1:
The patent segments computer program instructions into execution vectors (those that may generate hyperlinks) and non-execution vectors (those that do not). The static analysis tool divides the codebase into functional segments, identifying only those segments containing calls to external entities that could generate hyperlinks. This segmentation allows the crawler to execute only relevant code portions, improving performance while maintaining hyperlink discovery capability.
Solution Approach 2:
The patent applies local quality by treating different code segments differently based on their potential to generate hyperlinks. Rather than uniformly executing all code, the system identifies specific locations (execution vectors) where hyperlink generation may occur and executes only those segments. This localized execution approach optimizes resource allocation by focusing computational effort where it is most likely to yield results.
2Loss of information
If all computer program instructions are executed during crawling, then complete coverage of dynamic content is achieved, but computational resources are wasted on instructions that do not yield hyperlinks
Solution Approach 1:
The patent extracts and isolates execution vectors from the broader codebase through static analysis. By identifying and extracting only those code segments that contain calls to external entities capable of generating hyperlinks, the system removes unnecessary computational overhead. This extraction process separates useful information (potential hyperlink sources) from useless information (rendering and validation code), enabling efficient resource utilization while maintaining complete coverage of dynamic content where hyperlinks may be generated.
Data Source
AI summary
Crawling computer-based documents by performing static analysis on a computer-based document to identify within the computer-based document one or more execution vectors, where each execution vector includes a computer program segment including a call to an entity that is external to the computer-based document, and one or more additional computer program segments whose execution precedes and leads ultimately to execution of the computer program segment that includes the call to the entity, and causing any of the computer program segments in any of the execution vectors to be executed during a crawling of the computer-based document, and any computer program segment within the computer-based document that is excluded from the execution vectors to be excluded from execution during the crawling of the computer-based document.