Batch Rendering System Using Virtual Clock and Fetch Server
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Efficiently rendering a large number of web pages in real time is challenging due to the complexity of dynamically created pages with embedded items like images and scripts, which require significant resources and lead to rendering errors and slow processing times.
Innovation Solution
A batch rendering system comprising a rendering server and a fetch server, where the rendering server uses a virtual clock to manage tasks and the fetch server applies rewrite rules to avoid duplicative content fetches, storing dimensions instead of actual images, and using mock images to reduce processing load.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a batch rendering system renders a large number of web pages with embedded items, then the completeness of rendering results improves, but the processing time increases and rendering errors occur
Solution Approach 1:
The system pre-fetches embedded items (images, scripts, stylesheets) before the main rendering process. The fetch server proactively retrieves these resources in advance and stores them in a data store, so they are ready when needed during rendering, eliminating waiting time and preventing time-out errors.
Solution Approach 2:
A fetch server acts as an intermediary between the batch rendering system and external content sources. It handles all fetch requests, manages a data store of embedded items, and provides content to rendering engines, isolating the rendering process from network delays and errors.
2Measurement precision
If the fetch server fetches all embedded items for batch rendering, then the accuracy of rendering results improves, but the number of crawl requests increases and resource consumption increases
Solution Approach 1:
The system performs preliminary analysis of web pages to identify embedded items that are actually needed for rendering. The fetch server pre-fetches only these necessary items based on the page content and rendering requirements, avoiding unnecessary fetch requests for items that won't be used.
Solution Approach 2:
The system changes the state of embedded item content from external URLs to locally stored content in the data store. By converting remote resource references into local cached content, the system eliminates repeated network requests and reduces the quantity of crawl operations needed.
3Reliability
If the system stores actual content of embedded items, then the quality of rendering output improves, but the storage requirements increase and processing load increases
Solution Approach 1:
The system extracts only the essential properties of embedded items (such as dimensions, file type, and metadata) from the actual content, storing these extracted properties in the data store while keeping the full content available on-demand or in a separate storage system. This reduces the storage burden while maintaining rendering quality.
Solution Approach 2:
Instead of storing full-resolution actual images, the system creates and stores simplified copy representations (such as thumbnails or dimension-only data) that are sufficient for layout and rendering purposes, reducing storage requirements while maintaining functional rendering quality.
4Measurement precision
If the rendering engine processes all embedded items in real time, then the accuracy of content discovery improves, but the processing speed decreases and time-out errors increase
Solution Approach 1:
The fetch server performs preliminary retrieval and validation of embedded item content before the rendering engine processes it. By pre-fetching and verifying content availability, the system ensures accurate content discovery while allowing the rendering engine to proceed without waiting for network responses, maintaining high processing speed.
Solution Approach 2:
The system implements continuous parallel processing where the fetch server continuously retrieves embedded items while the rendering engine continuously processes pages. Both operations run concurrently without blocking each other, maintaining continuous productive action and preventing time-out errors.
Data Source
AI summary
Implementations include a batch-optimized render and fetch architecture. An example method performed by the architecture includes receiving a request from a batch process to render a web page and initializing a virtual clock and a task list for rendering the web page. The virtual clock stands still when a request for an embedded item is outstanding and when a task is ready to run. The method may also include generating a rendering result for the web page when the virtual clock matches a run time for a stop task in the task list, and providing the rendering result to the batch process. Another example method includes receiving a request from a batch process to render a web page, identifying an embedded item in the web page, and determining, based on a rewrite rule, that the embedded item has content that is duplicative of content for a previously fetched embedded item.