Web-Scale Data Processing with Columnar Distributed Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems for crawling and indexing the web face significant challenges due to the vast amount of data, which exceeds the storage and processing capacity of most individual computing devices, making it expensive and resource-intensive, primarily limited to large corporations.
Innovation Solution
A distributed computing approach is adopted, where the crawling and indexing process is decoupled into manageable chunks, using a column-oriented database to store and process data, with implicit record IDs and a hierarchical aggregation method to unify data from different epochs into a global namespace, reducing storage costs and improving processing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If web-scale data is stored and processed on a single computer, then data integrity is maintained, but storage capacity and processing speed are insufficient
Solution Approach 1:
The patent divides the web indexing system into multiple independent computing devices, each handling a portion of the crawling and indexing tasks. The data is partitioned across multiple devices with each device maintaining its own data structures and processing pipelines, enabling scalable storage capacity while preserving data integrity through distributed architecture.
2Reliability
If data is crawled and indexed in real-time, then data freshness is improved, but processing time and resource consumption increase
Solution Approach 1:
The patent implements preliminary action by pre-processing and indexing data during off-peak hours or in advance of actual queries. The system maintains data freshness by periodically updating indexes and caching frequently accessed data, allowing rapid response to user queries without requiring real-time processing of all web data.
Solution Approach 2:
The crawling and indexing operations are performed periodically rather than continuously in real-time. The system schedules regular crawls of web pages and updates indexes at intervals, balancing data freshness with processing resource management. This periodic approach reduces peak processing loads while maintaining acceptable data currency.
3Ease of manufacture
If traditional row-oriented databases are used, then ease of operation is maintained, but storage efficiency and query performance degrade
Solution Approach 1:
The patent employs column-oriented storage where data is organized by columns rather than rows, optimizing storage efficiency and query performance for specific data access patterns. Each column is stored separately with its own data types and compression schemes, allowing the system to tailor storage characteristics to the specific needs of different data fields while maintaining simplified access through the database interface.
Data Source
AI summary
A system and method for obtaining and processing web-scale data are provided herein. More particularly, a web-scale data processing system and method for crawling, storing, processing, encoding, and/or serving web-scale data are disclosed.