Web Page Principal Content Extraction via Candidate Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods fail to consistently extract important components like article titles, authors, and publication dates from complex Web pages, especially those with international content and multipage articles, due to rich layouts that interfere with focused reading.
Innovation Solution
A system that identifies and classifies items on a Web page, calculates candidate scores based on paragraph and image counts, and performs cleanup processing to select and reformatted the principal content, using formulas to prioritize relevant content and remove distractions like floating elements and navigation panels.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If rich Web layouts with multiple content elements are used, then Web page functionality and information density are improved, but readability and focused reading of principal content deteriorate
Solution Approach 1:
The system segments the Web page into multiple candidate regions and evaluates each independently using scoring formulas. This allows the rich layout to be processed in manageable units, identifying principal content without being overwhelmed by the overall complexity of the page structure.
Solution Approach 2:
The system extracts the principal content from the complex Web page layout by calculating scores for different candidate regions and selecting the top-scoring content. This extraction process separates the important content from distracting elements like navigation panels and advertisements, delivering a clean view to the user.
2Ease of manufacture
If existing content extraction methods are used, then simple Web pages can be processed, but consistent extraction of important components from complex Web pages deteriorates
Solution Approach 1:
The system changes the evaluation parameters by using multiple scoring formulas that consider different aspects of content quality (text density, image content, structural elements). This multi-parameter approach ensures consistent extraction across diverse Web page types, from simple articles to complex layouts with international content and multipage articles.
Solution Approach 2:
The system uses feedback mechanisms by comparing candidate scores and iteratively selecting the best content. The scoring system provides feedback on content quality, allowing the system to adjust and improve extraction consistency across different Web page configurations.
3Adaptability or versatility
If multiple content elements are included on Web pages, then functional completeness is improved, but interference with focused reading increases
Solution Approach 1:
The system converts the harmful effect of rich Web layouts into a benefit by using the structural information and content elements as scoring criteria. The very elements that cause reading interference (multiple content types, complex layouts) provide the data needed to identify and extract the principal content through the scoring formulas.
Solution Approach 2:
The system extracts only the principal content from the functionally complete Web page, removing distracting elements while preserving the essential information. This extraction delivers a focused reading experience while maintaining access to all functional elements on the original page if needed.
Data Source
AI summary
Extracting principal content from Web pages includes identifying and classifying items on the Web page, building a list of candidates, calculating candidate scores, selecting a top score candidate, performing clean up processing for the top score candidate, and performing final page processing for the top score candidate. Candidate scores may vary according to a number of paragraphs and images grouped according to size. A word length of CJK (Chinese-Japanese-Korean) text may be determined according to punctuation therein. Candidate scores may be modified according to a number of containers and pieces and wherein a container is a Web page element that is associated with tags ‘body’, ‘div’, ‘td’, ‘li’, ‘article/section’ and pieces are candidates that do not include other candidates. Candidate scores may be modified according to a number of ratios corresponding to text and link density.