Web Caching Supplement for Indexing Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Web crawlers face significant resource challenges and potential website blocking due to the need for frequent visits to update internet indexes, especially when websites customize content based on browser and location, leading to increased bandwidth usage and reduced accuracy in indexing.
Innovation Solution
Supplementing web crawler data with cached content from internet browser caches of volunteer panelists' devices, using on-device meters to periodically transmit cached internet content to a monitoring data collection site, which reduces the number of website visits required without compromising accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If web crawlers frequently visit websites to update indexes, then indexing accuracy is improved, but resource consumption increases and websites may block crawlers
Solution Approach 1:
The system performs preliminary actions by having web browsers cache website content locally before the crawler needs it. User devices pre-store website data during normal browsing, so when the crawler requests content, it can retrieve pre-cached versions without making frequent requests to websites, thus maintaining indexing accuracy while reducing resource consumption.
Solution Approach 2:
The invention introduces user devices with web browser caches as intermediaries between the web crawler and websites. Instead of the crawler directly querying websites frequently, it retrieves content from user device caches, which act as buffer layers. This intermediary approach reduces direct crawler-website interactions, lowering resource consumption while preserving indexing capabilities.
2Adaptability or versatility
If web crawlers make multiple visits with different browsers and locations, then content customization coverage is improved, but bandwidth usage increases
Solution Approach 1:
The system creates copies of website content stored in user device caches across multiple devices with different browsers and locations. Instead of the crawler making multiple visits to generate customized content views, it retrieves pre-existing copies from various user devices, maintaining content customization coverage while significantly reducing bandwidth consumption.
Solution Approach 2:
User devices serve multiple functions: they are both normal web browsers for users and data sources for the crawler. The same cached content on user devices fulfills both user browsing needs and crawler indexing requirements, eliminating the need for separate crawler infrastructure and reducing overall bandwidth usage.
3Reliability
If web crawlers continuously monitor website updates, then index freshness is improved, but the frequency of visits increases causing potential blocking
Solution Approach 1:
User devices automatically maintain local caches of website content during their normal browsing activities without requiring crawler intervention. This self-service caching mechanism continuously updates local copies as users browse, ensuring index freshness is maintained through user-driven updates rather than crawler-driven requests, thereby avoiding website blocking.
Data Source
AI summary
Methods and apparatus to supplement web crawling with cached data from distributed devices are disclosed. An example apparatus includes a browser monitor to extract a language setting from a web browser of a panelist computing device, and a data communicator to transmit data associated with a webpage accessed by the web browser and an indication of the language setting to a server.