Merging Continuous Event Processing with Map-Reduce for Real-Time Big Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional processing mechanisms struggle to handle large volumes of Big Data in real-time, as they are inefficient in producing immediate query results, especially in scenarios requiring up-to-the-minute insights, such as fraud detection or sales optimization, due to the limitations of relational databases and traditional Map-Reduce implementations.
Innovation Solution
Combining continuous event processing (CEP) with the Map-Reduce algorithmic tool to merge real-time CQL query results with batch processing results, allowing for continuous execution of CEP queries on newly arrived events while a Map-Reduce job processes stored data, thereby producing unified query results in real-time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If conventional relational databases and traditional Map-Reduce implementations are used, then data storage and batch processing are achieved, but real-time query result production is inefficient and time-consuming
Solution Approach 1:
The patent merges continuous event processing (CEP) with Map-Reduce batch processing into a unified system. The CEP component continuously processes streaming events in real-time, while the Map-Reduce component periodically processes accumulated data batches. The results from both components are merged and presented together, enabling simultaneous real-time responsiveness and comprehensive batch analysis without sacrificing either capability.
Solution Approach 2:
The patent segments the data processing system into distinct continuous event processing and batch processing components. Each component handles specific types of processing tasks independently - CEP handles immediate streaming events while Map-Reduce handles accumulated data batches. This segmentation allows both processing modes to operate in parallel without interfering with each other, improving overall real-time productivity.
2Speed
If continuous event processing is applied to large volumes of data, then real-time insights are produced, but processing speed and performance degrade due to data volume
Solution Approach 1:
The patent implements periodic batch processing alongside continuous event processing. Instead of continuously processing all accumulated data in real-time (which would slow down processing speed), the system periodically triggers Map-Reduce jobs to process accumulated data batches. This periodic action maintains fast real-time processing speed for immediate events while still handling large data volumes through scheduled batch processing.
Solution Approach 2:
The patent applies partial processing by dividing data handling into two parts: CEP processes only the most recent streaming events immediately, while Map-Reduce processes the accumulated historical data batches periodically. This partial action approach ensures that real-time processing speed is not degraded by the total data volume, as each component handles only its designated portion of the data workload.
3Measurement precision
If batch processing is performed on large datasets, then comprehensive analysis is achieved, but the time lag between data occurrence and analysis increases
Solution Approach 1:
The patent merges the results from continuous event processing and batch processing into a unified output. The CEP component provides immediate real-time results for recent events, while the Map-Reduce component provides comprehensive batch analysis results. By merging these two result streams, the system achieves both measurement precision from comprehensive batch analysis and minimal time lag from real-time processing.
Solution Approach 2:
The patent introduces a result merging mechanism that acts as an intermediary between the CEP and Map-Reduce components. This intermediary combines the real-time results from CEP with the comprehensive batch results from Map-Reduce, presenting a unified view that maintains measurement precision while minimizing perceived time lag for users.
4Adaptability or versatility
If traditional processing mechanisms are used, then system complexity is low, but the ability to handle both real-time and batch processing simultaneously is insufficient
Solution Approach 1:
The patent creates a universal processing system that handles both continuous event processing and batch processing through a unified architecture. The system uses a single event stream that can be processed by either CEP for real-time results or fed into Map-Reduce for batch processing, depending on the configured triggers and parameters. This multi-functionality allows the system to adapt to different processing requirements without requiring separate independent systems.
Data Source
AI summary
Large quantities of data can be processed and/or queried relatively quickly using a combination of continuous event processing and a Map-Reduce algorithmic tool. The continuous event processor can continuously produce real-time results by merging (a) CQL query results from events received since a currently executing Map-Reduce job was started with (b) a most recent query result produced by a most recently completed Map-Reduce job. When the currently executing Map-Reduce job completes, its query result can be stored and made accessible to the continuous event processor, and a new Map-Reduce job can be started relative to event data that has grown in size since the execution of the last Map-Reduce job. The Map-Reduce algorithmic tool provides a convenient mechanism for analyzing and processing large quantities of data.