Hybrid Data Processing for Real-Time Queries at Scale
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing systems face challenges in handling large volumes of data from various sources, as they struggle to provide real-time updates and maintain accurate information due to the vastness of data, making it difficult to query and maintain, especially with new information continuously becoming available.
Innovation Solution
A system that combines batch and real-time data processing capabilities, allowing for the reception and processing of both bulk and intermittent data inputs, using a common processing framework to generate summary data by formatting unstructured data into structured sets, consolidating information, and updating the data store based on differences between old and new data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If all web pages and data about businesses are stored in data stores, then the system has comprehensive information available, but the data becomes too large to be queryable in real-time
Solution Approach 1:
The patent divides the large-scale data processing system into two segments: a batch processing system that handles comprehensive data storage and initial processing, and a real-time processing system that handles query operations. The batch system pre-processes and structures data from all web pages and business information, while the real-time system queries this pre-processed data, enabling both comprehensive data retention and fast query responses.
Solution Approach 2:
The batch processing system performs preliminary actions by pre-processing, structuring, and organizing data before real-time queries are executed. This includes crawling web pages, extracting business information, and storing it in a structured format in advance, so that real-time queries can operate on already-prepared data without processing the entire dataset each time.
2Reliability
If the system processes all data continuously, then real-time accuracy is maintained, but the computational resources and time required become excessive
Solution Approach 1:
The system implements periodic batch processing alongside continuous real-time processing. The batch processing occurs at intervals to re-evaluate and update the structured data from all sources, while real-time processing continuously handles queries. This periodic re-processing maintains data accuracy without requiring continuous full-data processing, reducing computational overhead.
3Adaptability or versatility
If new information is continuously added to the data store, then the system remains up-to-date, but the complexity of maintaining and querying the data increases
Solution Approach 1:
The system segments data update operations into batch updates and real-time updates. New information is initially received and held in a staging area, then incorporated into the structured data through batch processing at appropriate intervals. This segmentation allows the system to remain adaptable to new information while managing complexity by processing updates in controlled batches rather than immediately integrating everything into the main queryable structure.
4Measurement precision
If the system reevaluates all data periodically, then comprehensive accuracy is achieved, but the processing time and resources increase
Solution Approach 1:
The batch processing system performs partial reevaluations by focusing on specific subsets of data or specific attributes that require verification, rather than completely reprocessing all data each time. This partial action approach maintains measurement precision for critical data points while reducing overall processing time and resource consumption compared to full reevaluation.
Data Source
AI summary
A traditional data processing system is configured to process input data either in batch or in real-time. On one hand, a batch data processing system is limiting because the batch data processing often cannot take into account any data received during the batch data processing. On the other hand, a real-time data processing system is limiting because the real-time system often cannot scale. The real-time data processing system is often limited to dealing with primitive data types and/or a small amount of data. Therefore, it is desirable to address the limitations of the batch data processing system and the real-time data processing system by combining the benefits of the batch data processing system and the real-time data processing system into a single data processing system.