Distributed DOM Query for Hidden Web Feature Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional web crawlers rely on HTML analysis, which fails to detect website characteristics that are not apparent in the HyperText Markup Language (HTML), limiting the extraction of interesting or developer-relevant information.
Innovation Solution
A distributed data query system using headless browsers and the Document Object Model (DOM) to analyze webpage content by interacting with objects in the DOM tree, allowing for the identification of characteristics such as geometric data and form recognition, and storing this data in a database for scalable analysis across multiple servers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional web crawlers use HTML analysis to browse websites, then the crawling process is simple and fast, but important website characteristics such as geometric data and form recognition cannot be detected
Solution Approach 1:
The patent introduces headless browsers as intermediary tools between the crawler and websites. These browsers render pages and extract DOM data, acting as a mediator that bridges the gap between simple HTML parsing and complex feature detection, enabling geometric data and form recognition without requiring complex crawling infrastructure
Solution Approach 2:
The patent replaces traditional mechanical HTML parsing with DOM-based interaction systems. Instead of simply reading HTML text, the system interacts with the Document Object Model to extract semantic information, geometric characteristics, and form data, substituting basic text processing with sophisticated object-oriented data extraction
2Productivity
If multiple websites are analyzed in parallel using traditional methods, then processing speed increases, but the ability to extract hidden characteristics remains limited
Solution Approach 1:
The patent segments the website analysis task into independent units by creating a queue system that distributes DOM data to multiple exploration devices. Each device processes specific websites independently in parallel, maintaining both high throughput and deep analysis capability through distributed computation
Solution Approach 2:
The patent adds a distributed computation dimension to traditional single-node web crawling. By introducing multiple exploration devices that process DOMs in parallel across different servers, the system achieves both high productivity through parallelism and high measurement precision through sophisticated DOM-based feature extraction at each node
3Measurement precision
If detailed DOM data is collected and stored for later analysis, then measurement precision improves, but data storage requirements and processing time increase
Solution Approach 1:
The patent extracts only the necessary DOM data and characteristics from the full webpage content. Instead of storing complete HTML pages, the system extracts specific features such as geometric data, form types, and other identifiable characteristics, reducing data volume while maintaining high measurement precision for the target attributes
Data Source
AI summary
Performing content exploration includes obtaining an exploration script indicating page characteristics to explore and a definition of webpages to explore, locating DOMs required for exploration script based on the definition of webpages to explore, generating a queue of DOMs required for the exploration script, and distributing the DOMS in the queue of DOMS to a plurality of exploration devices. Each exploration device opens a DOM from the queue of DOMS in a web browser, executes the exploration script on the opened DOM to obtain DOM data for the opened DOM, and associates the generated DOM data with the opened DOM in a knowledge store.