Versioned Resource Directory Annotation for Web Crawler Data Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current information management systems face inefficiencies in processing and storing data, particularly in web crawling, where large volumes of dynamic content require significant computational resources and bandwidth, and existing solutions fail to effectively manage updates and reprocessing of data, leading to redundant data retrieval and storage costs.
Innovation Solution
A computer-implemented process that retrieves and processes information streams by annotating resource directories with version numbers, allowing for efficient storage and reprocessing of segments based on updated entities, using a consolidation box that applies a map-reduce mechanism to identify and update entities, and stores profiles in a repository for indexing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If web crawlers download and process all web content to ensure complete information retrieval, then information completeness is improved, but network bandwidth consumption and computational resources increase significantly
Solution Approach 1:
The patent implements differential crawling by identifying and processing only specific portions of web content that have changed. Instead of re-crawling entire web pages, the system selectively retrieves only the modified segments based on version number comparisons, thereby reducing network bandwidth consumption while maintaining information completeness.
Solution Approach 2:
The system performs preliminary processing by storing version numbers of previously crawled content and comparing them with current versions before full reprocessing. This preliminary check allows the crawler to determine whether full re-crawling is necessary or if selective updates suffice, reducing unnecessary network traffic and computational overhead.
2Measurement precision
If the system reprocesses all retrieved information whenever resource directories are updated, then information accuracy is improved, but processing time increases
Solution Approach 1:
The patent segments the reprocessing task by identifying and processing only those information segments associated with updated entities. The system divides the overall processing workload into individual entity segments, allowing selective reprocessing of only the affected portions rather than reprocessing all retrieved information, thus reducing processing time while maintaining accuracy for updated content.
Solution Approach 2:
The system applies partial reprocessing by performing accuracy verification only on segments corresponding to updated entities in resource directories. Instead of excessively reprocessing all information, the system applies the necessary processing action only where needed, optimizing the balance between information accuracy and processing time.
3Productivity
If the system stores all retrieved articles for future reprocessing, then reprocessing efficiency is improved, but storage costs increase
Solution Approach 1:
The patent extracts and stores only the essential metadata (version numbers, entity identifiers, and segment pointers) from retrieved articles rather than storing complete copies of all articles. This extraction approach enables efficient reprocessing by providing sufficient information to locate and reprocess only necessary segments without occupying excessive storage capacity.
Solution Approach 2:
The system performs preliminary extraction of version numbers and entity associations during the initial crawling phase, storing this metadata for future use. This preliminary action prepares the data in a format that enables efficient selective reprocessing without requiring storage of complete article copies, thus improving reprocessing efficiency while minimizing storage requirements.
Data Source
AI summary
The invention is directed to a computer-implemented process for processing information related to entities and said entities being contained in a stream of information, the entities being stored in resource directories of a system, each resource directory containing entities and being annotated with a version number modified with the addition of at least one new entity. The process comprises the steps of: retrieving a stream of information from different information sources; processing segments of information from the stream of information in accordance with entities of said resource directories; storing the segments of information with an annotation indicating which version of said resource directories was used for processing said information segment; updating at least one of the said resource directories with at least one new entity and updating the version number of said at least one resource directory; and—reprocessing the stored segments of information containing the annotation indicating said segments of information were processed with a previous version of the at least one resource directory.