Multimodal Web Extraction Across Diverse Page Layouts
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing web information extraction methods struggle to efficiently convert schema-less semi-structured web pages to a structured format across multiple websites with differing designs and presentations, often relying on a single modality which is insufficient for accurate interpretation.
Innovation Solution
A neural network-based model using a transformer encoder with multi-modal embeddings and self-supervised learning to extract information from web pages, incorporating visual, text, and layout modalities, and employing a two-phase training process to learn inter-modality relationships.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a single modality is used for web information extraction, then the system complexity is reduced, but the accuracy and reliability of information extraction deteriorates
Solution Approach 1:
The patent combines multiple modalities (visual, text, layout, and markup) into a unified neural network model that processes all modalities simultaneously. The model integrates embeddings from different modalities to jointly represent web page elements, enabling accurate information extraction by leveraging complementary information from each modality rather than relying on a single modality alone.
2Adaptability or versatility
If traditional extraction methods are used on websites with differing designs, then the method can be applied broadly, but the extraction accuracy and efficiency deteriorates
Solution Approach 1:
The patent creates a universal information extraction model that can process web pages from multiple websites with different designs and presentations. The model uses multi-modal embeddings to capture both content and structural information, enabling it to adapt to various website layouts and design patterns while maintaining high extraction accuracy across diverse domains.
Solution Approach 2:
The patent adds layout position information as an additional dimension to the extraction process. By incorporating spatial coordinates and relative position relationships of web page elements, the model gains a new dimension for understanding page structure, which helps it adapt to different website designs while maintaining accurate information extraction.
3Measurement precision
If extensive labeled data is used for training, then the model accuracy improves, but the training time and resource requirements increase
Solution Approach 1:
The patent employs a two-phase training approach where the first phase uses self-supervised learning on large amounts of unlabeled data to pre-train the model. This preliminary action allows the model to learn useful representations from abundant unlabeled web pages. In the second phase, the model is fine-tuned on a smaller amount of labeled data, significantly reducing the time and resources needed compared to training from scratch with labeled data only.
Solution Approach 2:
The patent implements self-supervised learning where the model learns from unlabeled data by creating its own supervision signals. The model predicts properties of web page elements based on multi-modal inputs without requiring manual labels, enabling it to leverage vast amounts of unlabeled web pages for training and improving accuracy while minimizing the need for time-consuming labeled data preparation.
Data Source
AI summary
A machine learning model for extracting information from web pages is prepared. The preparation includes generating respective representations of a first set of web pages, including embeddings from screenshots and bounding boxes of the web pages for multi-phase training of the model. In a first phase of training of the model, multiple loss functions associated with respective prediction tasks are optimized jointly, including a markup language element prediction task and a prediction of overlap between bounding boxes and screenshot subdivisions. In a second phase of training, using output of a hidden layer of the model (whose parameters were learned in the first phase) as input, a loss function is optimized to achieve a target web information extraction objective. The trained version of the model is stored.