Event Driven Data Processing System for Near Real-Time Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing systems face inefficiencies in resource usage, time, and cost due to the lack of coordination and duplication in processing events, particularly in batch processing systems where issues are identified late and cannot be addressed in real-time.
Innovation Solution
An event-driven data processing system that includes an event queue, router, contextualizers, and a streaming component, which processes events in near real-time by routing them to appropriate context queues, de-duplicating events, and using a redrive component to update events in near real-time, thereby reducing resource waste and enabling immediate action on data changes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If batch processing is used, then resource usage and cost are reduced, but processing speed and real-time responsiveness deteriorate
Solution Approach 1:
The system segments the batch processing workload into individual event processing units that can be handled independently. Each event is routed to specific contextualizers based on event type, allowing parallel processing of multiple events simultaneously while maintaining resource efficiency through selective processing paths.
Solution Approach 2:
The system dynamically adjusts processing resources based on event types and priorities. The event router directs events to appropriate contextualizers, and the system can scale resource allocation dynamically, processing high-priority events in near real-time while lower-priority events are handled during off-peak periods, optimizing both speed and resource usage.
2Speed
If real-time data processing is implemented, then processing speed and responsiveness are improved, but resource usage and cost increase
Solution Approach 1:
The system applies different processing qualities to different events based on their specific requirements. Critical events receive immediate real-time processing with high resource allocation, while non-critical events are processed with lower resource intensity or deferred to batch processing, optimizing the balance between speed and resource consumption.
Solution Approach 2:
The system implements partial real-time processing by handling only the necessary portion of events in real-time through the event router and contextualizers, while other events can be processed in batches. This selective approach provides real-time responsiveness where needed without the full resource cost of universal real-time processing.
3Device complexity
If event duplication and lack of coordination occur, then system simplicity is maintained, but processing efficiency and resource utilization deteriorate
Solution Approach 1:
The event router serves multiple functions: it receives events from various sources, determines event types, routes events to appropriate contextualizers, and coordinates processing across the system. This multi-functional component provides coordination without requiring complex point-to-point communication between all system elements, maintaining relative simplicity while improving efficiency.
Solution Approach 2:
The event router acts as an intermediary between event sources and contextualizers, coordinating event distribution and preventing duplication. This central coordination point ensures that each event is processed by the appropriate contextualizer without redundant processing, improving efficiency while adding only a single coordinating component rather than complex distributed coordination logic.
4Device complexity
If issues are identified late in batch processing, then processing simplicity is maintained, but time to detect and correct issues deteriorates
Solution Approach 1:
The event router performs preliminary classification and routing of events before they are processed by contextualizers. By identifying event types and routing them appropriately in advance, the system detects potential issues early in the processing pipeline, allowing for immediate correction rather than waiting until batch processing completion.
Solution Approach 2:
The system implements feedback mechanisms where contextualizers can identify issues with events and route them back for correction. This feedback loop allows issues to be detected and addressed in near real-time during the processing flow, rather than waiting for batch processing to complete, improving issue detection time while maintaining processing clarity through structured feedback paths.
Data Source
AI summary
An event driven data processing system is disclosed that comprises event generators that generate events, an event queue that receives the events from the event generators, and an event router that receives the events from the event queue, and, for each event, selects a contextualizer based on an event type and transmits the event to a corresponding context queue associated with the selected contextualizer. The system also comprises context queues that receive the events from the event router and a plurality of contextualizers that receive the events from the context queues and, for each event, access context data sources, obtain additional context data, create a supplemented event, and store each supplemented event in an event datastore. The system further comprises a streaming component that streams each supplemented event from the event datastore for a period of time.