Webpage Text Extraction via Anchor-Based DOM Templates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing search systems face difficulties in automatically identifying and extracting relevant text content from webpages due to the mixing of useful content with non-relevant sections such as advertisements, leading to inaccurate summaries in search results.
Innovation Solution
The method involves accessing resources within a specific domain, identifying anchor patterns, generating a domain template to specify the location of relevant text content, and traversing the DOM path to extract and promote text content that is proximate to anchor text, thereby improving snippet relevance and including additional information from child DOM nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If automatic text content extraction is performed on webpages, then search results can be generated, but the accuracy of text content identification deteriorates due to mixing of useful content with advertisements and unrelated sections
Solution Approach 1:
The webpage is segmented into multiple sections based on visual characteristics and structural patterns. The system identifies and separates useful text content from advertisements and unrelated sections by analyzing DOM tree structure, anchor text patterns, and visual element characteristics. This segmentation enables accurate extraction of relevant content while maintaining processing efficiency.
Solution Approach 2:
The patent introduces an intermediary analysis layer that examines the relationship between visual elements and text content through anchor text and DOM tree structures. This intermediary layer mediates between the raw webpage content and the final extracted text, filtering out unrelated content while preserving useful information through pattern recognition and contextual analysis.
2Ease of operation
If text content is extracted from webpages with mixed content sections, then summary generation is possible, but the relevance of extracted text to the subject matter deteriorates due to inclusion of advertisements and unrelated content
Solution Approach 1:
The system applies local quality analysis by examining the specific characteristics of different webpage sections. It identifies regions with high relevance to the subject matter through analysis of anchor text patterns, DOM tree structures, and visual element properties. This allows the system to extract text content with high local relevance while maintaining the ability to generate comprehensive summaries.
Solution Approach 2:
The patent implements feedback mechanisms that continuously refine the text extraction process. The system analyzes the extracted text against the original webpage structure and anchor text patterns, adjusting the extraction criteria to improve relevance. This feedback loop ensures that only text content with high relevance to the subject matter is included in the final summary.
3Quantity of substance
If comprehensive text content is extracted from webpages, then more information is available for search results, but the accuracy of identifying useful text content deteriorates due to volume of mixed content
Solution Approach 1:
The system performs partial extraction by focusing on specific high-value regions of the webpage identified through pattern recognition. Rather than processing all content uniformly, it concentrates on sections with characteristic patterns of useful text content, such as those linked by anchor text to relevant resources. This partial action approach maintains high identification accuracy while reducing processing overhead.
Solution Approach 2:
The patent applies preliminary analysis of DOM tree structures and anchor text patterns before full text extraction. This preliminary action identifies and prioritizes sections likely to contain useful text content, allowing the system to extract comprehensive information from the most relevant sections first, thereby maintaining high accuracy even as the volume of extracted content increases.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for determining summary content for resources in a domain. In one aspect, a method includes accessing a first resource belonging to a particular domain, selecting an anchor in the first resource linking to a second resource belonging to the particular domain, identifying particular text content in the first resource that is subordinate to the anchor that the second resource includes the particular text content that is subordinate to the anchor, based on determining that the second resource includes the particular text content that is subordinate to the anchor, generating a domain template for the particular domain, the domain template specifying a location of the particular text content in the second resource, and determining, for each respective resource belonging to the particular domain having a structure matching the domain template, respective text content for the respective resource.