Web Knowledge Extraction via Auto Wrapper Structuring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional search engines face challenges in providing users with relevant information due to lengthy and noisy search result lists, making it difficult for users to find desired information effectively.
Innovation Solution
The technique involves generating structured information from semi-structured web pages through an auto wrapper solution that is noise-tolerant and scalable, using document object model (DOM) transformations to extract tag path text and position vectors, which are then used to determine root and detail templates, and storing this information in a knowledge base for efficient retrieval in response to user queries.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If traditional search engines provide comprehensive search result lists, then information completeness is improved, but search result length and noise increase making it difficult for users to find desired information
Solution Approach 1:
The patent segments web pages into structured data elements by extracting specific information fields (e.g., product names, prices, specifications) from unstructured or semi-structured content. This segmentation transforms comprehensive but noisy search results into organized, discrete data points that can be quickly scanned and compared, maintaining information completeness while reducing user search time.
Solution Approach 2:
The system performs preliminary extraction and structuring of information from web pages before the search query is executed. By pre-processing web content into structured formats during offline crawling, the system prepares data in advance so that when users submit queries, the results can be immediately retrieved and presented in organized form without requiring users to parse through unstructured content during the search process.
2Loss of information
If search engines provide detailed search results, then information completeness is improved, but result complexity and noise increase
Solution Approach 1:
The patent applies local quality by extracting and structuring specific fields (price, rating, specification) from different regions of web pages according to their semantic meaning. Each extracted element is tagged with its specific type and position, creating locally optimized data structures that maintain detailed information while presenting it in a uniformly simple format. This allows the system to preserve information detail without increasing overall result complexity.
3Extent of automation
If automated information extraction processes web pages, then structured information generation is improved, but processing time and computational resources increase
Solution Approach 1:
The system performs automated information extraction as a preliminary action during offline web crawling, before online search queries are processed. By pre-extracting and structuring data from web pages during idle periods, the system maintains high automation levels while ensuring that the actual search operation can quickly retrieve and present results without incurring extraction delays during user interactions.
Solution Approach 2:
The patent creates simplified copies or representations of web page content in structured formats (such as JSON or database records) that preserve essential information while reducing complexity. These copied structured representations are stored for quick retrieval, allowing the system to maintain high automation in information extraction while significantly reducing the time required to process and present search results.
Data Source
AI summary
Techniques are described for generating structured information from semi-structured web pages, and retrieving the structured knowledge in response to a user query that indicates a query intent. The structured information is automatically extracted offline from semi-structured web pages, through the use of an auto wrapper solution that is noise tolerant, scalable, and automatic. The structured information is stored in a knowledge base, and provided in response to a user search query that indicates a query intent. Extraction of structured information may also include clustering of pages based on their measured similarities. The clusters may be determined based on similar elements in the tag path text data of the pages. A minimum size threshold may be applied to the clusters.