Web Page Rendering Engine Off-Line Resource Scheduling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Efficiently rendering a large number of web pages for indexing without flooding servers with simultaneous requests for embedded resources, as web browsers can do in real-time but web page indexing processes cannot.
Innovation Solution
A system comprising a web crawling engine, a scheduling engine, and a rendering engine that crawls web pages, stores content and crawl times, and schedules rendering to avoid real-time resource requests, ensuring each resource is crawled exactly once and stored for subsequent rendering.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a rendering engine attempts to render thousands or tens of thousands of web pages that embed the same external resource at the same time, then the rendering speed and productivity are improved, but the server on which the external resource resides would be flooded with near simultaneous requests
Solution Approach 1:
The system performs preliminary crawling and storage of external resources before the rendering phase. The web crawling engine crawls and stores external resources (images, JavaScript, style sheets) in a data repository in advance, so that during rendering, these resources are already available locally without requiring real-time server requests. This eliminates server overload while maintaining high rendering productivity.
Solution Approach 2:
The patent introduces a data repository as an intermediary between the web crawling engine and the rendering engine. This repository stores external resources locally, acting as a buffer that decouples the rendering process from external resource servers. The rendering engine retrieves resources from this local repository rather than making direct real-time requests to external servers, thus preventing server flooding while enabling parallel rendering.
2Manufacturing precision
If a web browser renders web pages in real time, then the rendering accuracy is improved, but it is not feasible for a web page indexing process to render a large number of pages in real time
Solution Approach 1:
The system segments the web page indexing process into three distinct phases: (1) web crawling phase where URLs and external resources are collected and stored, (2) scheduling phase where rendering tasks are queued and managed, and (3) rendering phase where pages are rendered to images. This segmentation allows each phase to be optimized independently, enabling high throughput in rendering while maintaining accuracy through complete resource availability.
Solution Approach 2:
The system performs preliminary crawling and storage of all external resources (images, scripts, stylesheets) before the rendering phase begins. This preliminary action ensures that during rendering, all resources are already available in the data repository, allowing the rendering engine to produce accurate rendered pages without real-time network dependencies, thereby achieving both high accuracy and high throughput.
3Reliability
If the same external resource is downloaded for each web page that embeds it, then the rendering completeness is improved, but it is neither feasible nor efficient for a rendering engine to do so for thousands of pages
Solution Approach 1:
The system creates local copies of external resources in a data repository during the web crawling phase. Instead of downloading the same external resource (such as a commonly used JavaScript library or style sheet) repeatedly for each web page, the system crawls and stores each unique external resource once, then reuses these stored copies during rendering. This ensures rendering completeness while eliminating redundant network transfers and energy consumption.
Solution Approach 2:
The system merges the web crawling function with the external resource collection function. The web crawling engine not only crawls web pages but also simultaneously crawls and stores all external resources (images, scripts, stylesheets) in the same data repository. This merging allows the rendering engine to access all necessary resources from a single local source, ensuring completeness while avoiding redundant downloads and reducing network energy consumption.
Data Source
AI summary
System and method for indexing rendered web page images. A web crawling engine stores the content and crawl time of a web page. A scheduling engine sends the content and crawl time to a rendering engine, and processes requests for embedded objects. If a requested object has been crawled, it sends the contents to the rendering engine. Otherwise it schedules the crawl of the object, and once the object is crawled, it resends the content and crawl time of the web page to the rendering engine. The rendering engine receives the content and crawl time of a web page, requests all embedded objects, and renders the web page to an image once all embedded objects are received.