Client-Side Lossless DOM Generation for Web Content Analysis

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Generating an accurate and lossless representation of dynamic webpages is challenging due to the dynamic nature of content, which leads to inefficient content checks and analysis, as existing methods fail to fully capture the rendered state of webpages, especially when analyzing at a remote server system.

Innovation Solution

A method and system for generating a lossless static object model of a dynamic webpage on a client device, incorporating all browser data sources, allowing direct ingestion and transfer of the Document Object Model (DOM) from the client to the server, bypassing internal rendering at the server, and enabling efficient content checks without re-rendering the webpage.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If static analysis of webpage files (HTML, scripts, CSS) is performed at a remote server, then the analysis process is simple, but it fails to fully capture the rendered state and dynamic content modifications

Engineering Contradiction:

Improveaccuracy of webpage representationVSAvoidcomplexity of analysis process

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

Instead of having the server render the webpage and generate the DOM (traditional approach), the invention inverts the process by having the client device render the webpage and generate the DOM locally, then transfer only the resulting DOM structure to the server for analysis. This resolves the contradiction by achieving accurate rendered-state capture without requiring the server to perform complex rendering operations.

Inventive Principle:

Principle #13The other way round (Inversion)

Solution Approach 2:

The invention introduces the client device's web browser as an intermediary that performs the rendering and DOM generation. The browser acts as a mediator between the webpage files and the server analysis system, capturing the full rendered state including dynamic content modifications, then passing this accurate representation to the server without requiring the server to perform rendering itself.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Measurement precision

If a webpage is dynamically rendered in a web browser at the client device, then an accurate representation of the rendered state is obtained, but exporting a static representation to the server is challenging and slow

Engineering Contradiction:

Improveaccuracy of webpage representationVSAvoidspeed of content check process

Core Design Contradiction:

Measurement precisionVSProductivity

Solution Approach 1:

The invention extracts only the essential DOM structure and rendered state data from the client device's browser and transfers it to the server, rather than transferring the entire rendered webpage or requiring re-rendering at the server. This extraction approach maintains accuracy while dramatically improving transfer speed and server processing efficiency.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The client device creates a copy of the DOM structure and rendered state data from the browser's memory and transfers this copy to the server. This copying mechanism allows the server to receive an accurate representation of the rendered webpage without needing to perform the rendering process itself, thus improving productivity while maintaining precision.

Inventive Principle:

Principle #26Copying

3Reliability

If the entire DOM is transferred from the client to the server for content checks, then complete webpage analysis is possible, but processing time and resource usage increase

Engineering Contradiction:

Improvecompleteness of webpage analysisVSAvoidprocessing time for content checks

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The invention segments the DOM transfer process by sending only the essential structural elements and rendered state data that are necessary for content checks, rather than transferring the entire DOM including all intermediate processing data. This segmentation maintains completeness of analysis while reducing processing time and resource usage.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The invention applies partial action by transferring only the specific DOM elements and attributes that are relevant for content checks and analysis, rather than transferring the complete DOM structure. This partial transfer approach maintains reliability of analysis for the required elements while significantly reducing processing time and resource consumption.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS11960561B2Client-side generation of lossless object model representations of dynamic webpages

Publication Date: 2024.04.16 SITEIMPROVE AS

AI summary

The disclosed technology provides for generating a lossless representation of a webpage on a client device. A method can include rendering, by a client device, a webpage to be presented in a web browser at the client device, receiving an indication of user input to run a content check on the rendered webpage, calling, in response to receiving the indication of user input, a scraping function to scrape a document object model (“DOM”) of the rendered webpage in the web browser at the client device, generating a lossless representation of the scraped DOM of the rendered webpage, and transmitting, to a web content analysis system, the lossless representation of the scraped DOM to be used when running the content check for the rendered webpage. The lossless representation of the scraped DOM can be a flat object model (“FlatOM”) of the DOM of the rendered webpage.