Template-Driven Web Page Parser for Declarative Data Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing parsers for extracting data from web pages and hierarchically structured documents are cumbersome and difficult to maintain, requiring extensive knowledge and complex navigation of DOM trees, making it challenging to efficiently extract data from structured documents.
Innovation Solution
A template-driven parser that uses declarative templates to specify pattern matches for subjects, predicates, and objects, allowing for the generation of lists of one-to-one relationships by applying XPath matching patterns, with options for omitting predicates, specifying annotations, and defining cardinality and scope, enabling easier configuration and data extraction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If traditional scrapers are used to extract data from web pages, then data extraction can be performed, but the scraper code generators are cumbersome and require extensive knowledge of the programming language
Solution Approach 1:
The patent uses template definitions that can be copied and reused to generate scraper code. Instead of writing complex scraper code from scratch, users can copy pre-defined templates and modify them to suit their needs, significantly reducing the knowledge and effort required while maintaining data extraction functionality
Solution Approach 2:
The patent introduces templates as an intermediary layer between the user and the actual scraper code generation. Users work with high-level template definitions rather than directly writing complex scraper code, and the system automatically generates the underlying scraper code from these templates, simplifying the user's task while handling the complexity automatically
2Productivity
If complex navigation of DOM trees is used to extract data from structured documents, then data extraction can be performed, but the process becomes difficult to maintain and requires extensive knowledge
Solution Approach 1:
The patent performs preliminary action by pre-defining templates that encapsulate complex DOM navigation logic. Instead of navigating complex DOM trees during actual data extraction, the navigation logic is prepared in advance through template definitions, making the extraction process simpler and more maintainable while maintaining efficiency
Solution Approach 2:
The patent allows copying of pre-defined template patterns for common data extraction scenarios. Rather than developing complex parsers from scratch for each task, users can copy and adapt existing templates, reducing development complexity and maintenance burden while achieving efficient data extraction
3Adaptability or versatility
If traditional parsing tools are used, then data can be extracted from web pages, but the tools lack ease of configuration and control
Solution Approach 1:
The patent introduces dynamic template definitions that can be flexibly configured and adapted to different data extraction needs. The templates support dynamic specification of subjects, predicates, objects, and relationships, allowing users to easily adapt the parsing configuration to various scenarios without being constrained by rigid tool structures
Solution Approach 2:
The patent creates a universal template system that can handle multiple types of data extraction tasks through a single configurable framework. The templates can define various relationships (one-to-one, one-to-many, many-to-many) and can be applied to different web page structures, providing both versatility and ease of configuration through a unified approach
Data Source
AI summary
The present invention relates to generating a parser for and to parsing a web page to generate a list of one-to-one relationships, such as parsing a publically available Twitter page to generate a list of followers and a list of photographs or a publically available list of what is “happening now at the Nation” on the music web site ReverbNation. More generally, it relates to a parser that takes parsing instructions from a declaratory template. The declaratory template used to generate lists can be as simple as specifying pattern matches for a subject, predicate and object. In alternative implementations, specification of a predicate could be omitted, if only one type of list were being generated. In other alternative implementations, a string user text can be specified. Another option is to specify annotations to relations, which can be literally annotated or extracted using a pattern match specification. Cardinality of the subject and object can be specified. Multiple declaratory templates can be used to extract multiple lists from the same web page. Query group statements can be used to set a scope in which subject and object pattern matches must occur in order to emit a relationship.