Event Data Aggregation for Real-Time Validation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data processing approaches face challenges such as data mutability, difficulty in processing large volumes of complex data, and the lack of real-time insights or actions, leading to inefficiencies and the need for more robust, immutable, traceable, and auditable data processing solutions.
Innovation Solution
A central server computer equipped with a real-time event streaming module, an aggregator module, and a workflow module, which processes event data packets in real-time, aggregates them according to transactions, validates the data, and stores it in an events database for further processing, ensuring data immutability and traceability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional data processing approaches are used, then system complexity is reduced, but real-time processing capability and data reliability deteriorate
Solution Approach 1:
The system divides data processing into distinct modular components: event streaming module, aggregation module, validation module, and storage module. Each module handles specific processing tasks independently, improving reliability through specialized processing while managing complexity through modular architecture. The segmentation allows each component to be optimized for its specific function while maintaining overall system coherence.
Solution Approach 2:
The patent introduces an aggregation module as an intermediary between event streaming and validation processes. This intermediary layer groups related events into packets before validation, enabling batch processing that enhances data reliability without requiring all validation operations to process every individual event in real-time, thus managing system complexity.
2Speed
If real-time processing is implemented, then processing speed is improved, but data validation and aggregation complexity increases
Solution Approach 1:
The aggregation module performs preliminary grouping of events into packets before they reach the validation module. This preliminary action organizes data in advance, reducing the complexity of validation operations during real-time processing. By pre-aggregating events, the system prepares data structures that simplify subsequent validation and processing steps.
Solution Approach 2:
The system merges multiple validation operations into a unified validation module that processes aggregated event packets collectively. This merging approach handles complex validation logic more efficiently by operating on grouped data rather than individual events, maintaining real-time processing speed while managing validation complexity through batched operations.
3Reliability
If data aggregation is performed, then data integrity is improved, but processing time increases
Solution Approach 1:
The aggregation module performs partial aggregation by grouping events into packets of appropriate size rather than aggregating all events simultaneously. This partial action maintains data integrity through structured grouping while minimizing processing time by avoiding excessive aggregation delays. The packetization approach allows incremental processing of aggregated data.
Solution Approach 2:
The system implements continuous event streaming and aggregation operations that run without interruption. The validation module continuously processes aggregated packets as they become available, eliminating idle time between aggregation and validation steps. This continuous operation ensures data integrity through consistent processing while minimizing overall processing time through eliminate gaps in the workflow.
Data Source
AI summary
A central server computer can process the data in real time by using an event-based data (i.e., event data) platform. Event-based data can be actions performed by entities providing the data to the central server computer. Storing the event-based data can allow the central server computer to store a transaction data in a granular manner according to different events, which can help with verification and traceability.