LLM-Guided Parsing Expressions for Changing Web Page Layouts
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current web scraping systems face inefficiencies in adapting to frequent changes in webpage layouts, requiring manual and resource-intensive methods to generate parsing expressions, and often result in inaccurate data extraction due to outdated models.
Innovation Solution
A system utilizing machine learning to generate parsing expressions by adding index values to webpage elements, interfacing with a large language model to identify elements, and generating expressions that can adapt to layout changes, allowing for efficient and accurate data extraction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual methods are used to generate parsing expressions, then accuracy can be maintained, but the process becomes time-consuming and resource-intensive
Solution Approach 1:
The system automatically generates parsing expressions by downloading web pages, extracting elements, and creating expressions without requiring manual intervention. The parser self-updates its expressions when webpage layouts change, eliminating the need for manual regeneration while maintaining accuracy.
Solution Approach 2:
The system pre-processes web pages by downloading them and extracting elements before generating parsing expressions. This preliminary action allows the system to have ready-made reference data when expressions need to be generated or updated, reducing overall processing time.
2Reliability
If outdated parsing expressions are used, then resource consumption is reduced, but data extraction accuracy deteriorates due to layout changes
Solution Approach 1:
The system continuously monitors webpage layouts and receives feedback about changes. When layout changes are detected, the system automatically updates parsing expressions based on the new layout structure, ensuring continued accuracy without requiring manual intervention or resource-intensive retraining.
Solution Approach 2:
The parsing expressions are made dynamic and adaptive rather than static. The system automatically generates new expressions when webpage layouts change, allowing the parser to adapt to evolving web structures while maintaining efficient resource consumption through automated updates.
3Adaptability or versatility
If frequent updates to parsing expressions are made to adapt to layout changes, then data extraction accuracy improves, but system complexity increases
Solution Approach 1:
The parser automatically detects layout changes and generates updated expressions without external intervention. The system self-manages the complexity of expression generation and updates, reducing the operational burden while maintaining high adaptability to webpage layout changes.
Solution Approach 2:
The system segments the web page into discrete elements and generates parsing expressions for each element independently. This segmentation allows the system to handle complex pages by processing them in manageable units, reducing overall system complexity while maintaining adaptability.
4Reliability
If multiple parsing expressions are generated and used, then data extraction reliability improves, but computing resources increase
Solution Approach 1:
The system generates multiple parsing expressions for robustness but uses them selectively based on success. Rather than always executing all possible expressions, the system tries expressions in sequence and stops when one succeeds, reducing overall resource consumption while maintaining high extraction reliability through the preparation of multiple expressions.
Data Source
AI summary
Disclosed herein are system, method, and computer program product embodiments for improving web scraping technology by using machine learning to generate parsing expressions. A system receives a request to identify an element in a first document at a target web page. The system downloads and modifies the first document by adding an index value as an attribute to a tag for the element. A query is submitted to a large language model (LLM), including the modified first document, a description of the element, and a request asking the LLM to identify the element based on the description. The system obtains, from the LLM, the index value assigned to the element. The system generates an expression defining a path to the element in the first document using the index returned by the large language model. The system downloads a second document, and parses data of a second element using the expression.