Localized Graph Analysis for Webpage Data Extraction Reliability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional techniques for extracting information from semi-structured documents, such as webpages, are unreliable when the structure changes, as they fail to determine whether modifications affect the reliability of the extracted information, requiring frequent manual updates of extraction rules.
Innovation Solution
An extraction module uses localized graph analysis to determine if changes to the structure of a document affect the reliability of extracted information, allowing for reliable extraction without human intervention by comparing subgraphs before and after modifications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional extraction techniques are used, then extraction rules are simple to implement, but reliability of extracted information deteriorates when document structure changes
Solution Approach 1:
The patent segments the document structure into a graph representation where elements are nodes and relationships are edges. This segmentation allows the system to analyze specific portions (subgraphs) of the document structure independently, enabling reliable information extraction by focusing on the relevant structural relationships rather than requiring complete structural matching.
Solution Approach 2:
The patent transforms the document structure into a graph representation with specific parameters (node types, edge relationships, hierarchical levels). By changing the representation parameters from traditional text-based or tag-based structures to graph-based structures with defined node and edge attributes, the system can more effectively detect structural changes while maintaining extraction reliability.
2Measurement precision
If manual updates of extraction rules are performed, then information extraction remains accurate, but labor costs and time increase
Solution Approach 1:
The patent implements a feedback mechanism that automatically detects changes in document structure by comparing current graph representations with previous versions. When structural changes are detected, the system automatically updates the extraction rules by identifying unchanged subgraphs and adapting query expressions, eliminating the need for manual rule updates while maintaining extraction accuracy.
Solution Approach 2:
The system performs self-service by automatically detecting document structure changes and updating extraction rules without human intervention. The graph-based representation enables the system to autonomously identify which parts of the document structure have changed and which remain stable, automatically adjusting extraction queries to maintain accuracy.
3Measurement precision
If frequent updates of extraction rules are performed, then information extraction accuracy is maintained, but productivity decreases
Solution Approach 1:
The patent performs preliminary action by pre-processing documents into graph representations and pre-identifying stable subgraphs that are unlikely to change. This preliminary structuring allows the system to quickly detect actual changes versus cosmetic modifications, reducing the frequency of necessary rule updates while maintaining extraction accuracy and improving overall productivity.
Data Source
AI summary
A webpage containing information to be extracted may undergo changes to a layout of elements that present the information. These changes could result in an inability to retrieve the information later. A first graph is determined that represents elements of a first version of a webpage at a first time. An element in the first graph for which information is being acquired is specified. A relevant portion of the first graph is designated that includes the element and immediate neighbors in the first graph. Later, a second version of the webpage is retrieved, and a second graph of that second version is determined. The relevant portion of the first graph is compared to the second graph. If a match is found, the information of interest is extracted from the specified element of the second graph. This allows extraction of information to proceed even if the layout of elements changes.