Server-Side User Agent Emulation for Web Scraping

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current Server-side JavaScript technologies fail to provide complete interactions characteristic of rich web sites and applications, particularly lacking browser-type functionality.

Innovation Solution

Emulation of multiple user agents on a server-side system that includes identifying and modifying user agent strings, parsing, and executing web pages as a browser would, allowing for full content access and optimization of web page loading by aggregating resources and caching frequently used code.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If server-side JavaScript technologies are used, then web page processing capability is improved, but browser-type functionality is lost

Engineering Contradiction:

Improveweb page processing capabilityVSAvoidbrowser-type functionality

Core Design Contradiction:

ProductivityVSAdaptability or versatility

Solution Approach 1:

The patent implements a virtual browser engine that copies and replicates browser functionality on the server side. This virtual browser can parse HTML, execute JavaScript, render CSS, and interact with web pages just like a real browser would, enabling server-side processing while maintaining browser-type capabilities. The virtual browser engine replicates core browser functions including DOM parsing, JavaScript execution, and rendering engines.

Inventive Principle:

Principle #26Copying

Solution Approach 2:

The server-side JavaScript technology is designed to perform multiple functions: it can process web pages, emulate browser behavior, parse HTML, execute JavaScript code, render CSS styles, and interact with various web technologies. This multi-functional approach allows a single server-side system to handle diverse web processing tasks that traditionally required separate tools or technologies.

Inventive Principle:

Principle #6Universality (Multi-functionality)

2Reliability

If full web pages are requested and served, then content completeness is improved, but bandwidth efficiency deteriorates

Engineering Contradiction:

Improvecontent completenessVSAvoidbandwidth efficiency

Core Design Contradiction:

ReliabilityVSLoss of energy

Solution Approach 1:

The patent extracts and processes only the necessary portions of web pages using the virtual browser engine. Instead of downloading and processing entire web pages, the system can selectively parse and execute specific JavaScript functions, render only necessary HTML elements, and extract required data from web pages. This extraction approach maintains content completeness for what is needed while significantly reducing unnecessary bandwidth consumption.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The system performs partial processing of web pages by executing only the necessary JavaScript code and rendering only the required content. The virtual browser engine can selectively process web pages based on specific requirements, such as executing only particular JavaScript functions or rendering only specific HTML elements, rather than processing the entire page unnecessarily.

Inventive Principle:

Principle #16Partial or excessive action

3Ease of manufacture

If web pages are reloaded completely on each interaction, then simplicity of implementation is improved, but user interface performance deteriorates

Engineering Contradiction:

Improvesimplicity of implementationVSAvoiduser interface performance

Core Design Contradiction:

Ease of manufactureVSProductivity

Solution Approach 1:

The patent implements preliminary processing of web pages using the virtual browser engine before they are served to clients. The system can pre-parse HTML, pre-execute JavaScript code, pre-render CSS styles, and pre-process content in advance. This preliminary action allows the server to prepare web pages in advance, reducing the need for complete reloads on subsequent interactions and improving user interface performance.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system segments web page processing into distinct components: HTML parsing, JavaScript execution, CSS rendering, and content generation. The virtual browser engine handles these segments separately and can return only the necessary processed content to clients. This segmentation allows for more efficient processing and enables partial page updates without requiring complete page reloads.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS9148467B1System and method for emulating different user agents on a server

Publication Date: 2015.09.29 APPCELERATOR

AI summary

A system and method for emulating a multitude of different user agents on a server-side is disclosed herein. On a server-side, an acceptable user agent string for receiving full content from the web-site with a browser sniffing agent is identified. The acceptable user agent is emulated by transmitting a HTTP request with the acceptable user agent string, allowing for access to the full content of the web-page of the web-site.