AI-Guided URL Discovery Actions for Fewer Web Recrawls
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing web discovery systems are resource-intensive and time-consuming due to analyzing outgoing URL links during recrawl cycles, leading to inefficiencies and unreliable URL discovery.
Innovation Solution
A discovery action system utilizing a generative AI model with prompt generation processes to partition URLs into classes, determine optimal discovery actions, and execute them efficiently, minimizing recrawls by leveraging contextual knowledge and previous actions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional web discovery systems analyze outgoing URL links during recrawl cycles, then new URLs can be discovered, but the process becomes resource-intensive and time-consuming
Solution Approach 1:
The system performs preliminary actions by using generative AI to predict and prioritize high-value target URLs before actual crawling occurs. Instead of randomly or systematically crawling all outgoing links, the AI model analyzes contextual information from seed URLs to pre-identify promising target URLs, thereby reducing the need for extensive recrawling and computational resources while maintaining high URL discovery productivity
2Reliability
If traditional web discovery systems perform extensive recrawls to discover new URLs, then more URLs can be found, but the process becomes unreliable and delays discovery
Solution Approach 1:
The patent replaces the mechanical systematic recrawling approach with an intelligent AI-based prediction system. The generative AI model substitutes the brute-force mechanical process of crawling all outgoing links with a cognitive system that understands web context, link patterns, and URL relationships, thereby improving discovery reliability and reducing time loss through smarter, more targeted URL identification
3Measurement precision
If web discovery systems crawl more URLs to increase discovery accuracy, then more new URLs are found, but the system exceeds web crawler traffic limits
Solution Approach 1:
The system applies local quality by concentrating crawling efforts on specific high-priority target URLs identified by the AI model rather than uniformly crawling all outgoing links. The generative AI analyzes contextual information to determine which specific URLs are most likely to yield valuable discoveries, allowing the system to achieve high discovery accuracy while minimizing total crawl volume and staying within traffic limits by focusing resources on the most promising local areas of the web graph
Data Source
AI summary
This disclosure describes a framework for generating improved uniform resource locator (URL) discovery actions for classes of URLs using a discovery action system. Specifically, this disclosure describes a discovery action system that utilizes a prompt generation process with a generative artificial intelligence (AI) model to efficiently generate optimal URL discovery actions for different URL classes. For instance, the discovery action system utilizes an iterative prompt generation process that incorporates previously generated discovery actions with a generative AI model to determine improved discovery actions for a specific URL class. These improved discovery actions are then used to determine new URLs for the class. In addition, once the optimal URL discovery actions are determined for a URL class, the discovery action system facilitates the discovery of new URLs for the URL class without relying on the generative AI model.