Real-Time Transaction Maps for Low-Latency Data Aggregation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data aggregation techniques face inefficiencies in processing high volumes of transactions due to state explosion, limited flexibility, high latency, and resource wastage in computing and network resources, especially when calculating and transmitting large amounts of data.
Innovation Solution
A method and system for data aggregation that utilizes a map data structure to store transaction data based on keys, allowing for efficient storage and calculation of aggregation values, with the ability to filter and encrypt data, and distribute time-based map structures across servers for improved processing and resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If current techniques proactively store the state of all variables for every predetermined time period, then data aggregation coverage is improved, but computing resources are wasted due to state explosion and storing unused states
Solution Approach 1:
The system performs preliminary filtering of events based on aggregation parameters before storage. Event filters are applied in advance to identify only relevant events for each aggregation, preventing the storage of unnecessary states and reducing computing resource consumption while maintaining comprehensive aggregation coverage.
Solution Approach 2:
Different filtering strategies and aggregation parameters are applied locally to different aggregation operations. Each aggregation can have its own customized filter criteria and time period settings, allowing the system to optimize resource usage for each specific aggregation while maintaining versatility across multiple aggregation types.
2Quantity of substance
If all incoming raw data is stored and aggregations are calculated upon request by searching through raw data, then data completeness is improved, but processing latency increases due to searching through large amounts of data
Solution Approach 1:
The system performs preliminary organization of raw data into aggregated structures with associated filters before requests are made. Aggregation results are pre-computed and stored in a searchable format, allowing rapid retrieval when requests are received without needing to search through all raw data, thus reducing latency while maintaining data completeness.
Solution Approach 2:
An intermediary aggregation layer is introduced between raw data storage and request processing. This layer pre-processes and organizes data according to aggregation parameters, serving as a mediator that enables fast retrieval of aggregation results without requiring direct searching of raw data, thereby reducing processing latency.
3Ease of operation
If portions of raw data are transmitted to user devices for local calculation, then user device control is improved, but network resources are wasted due to transmission of voluminous information
Solution Approach 1:
The system extracts and transmits only the essential aggregation parameters and filtered event data needed for user device calculations, rather than transmitting complete raw data sets. This selective extraction reduces network transmission volume while still enabling user device control and local calculation capabilities.
4Device complexity
If aggregation parameters and time periods are predetermined, then system complexity is reduced, but user flexibility is limited in specifying custom aggregation time periods
Solution Approach 1:
The system dynamically adjusts aggregation parameters and time periods based on user requests and event characteristics. Rather than using fixed predetermined parameters, the system can adaptively modify aggregation windows, filters, and time periods to match specific user needs and event patterns, maintaining low complexity through automated adaptation while providing high user flexibility.
Data Source
AI summary
Provided is a method for aggregating data from real-time events (e.g., payment transactions). The method may include receiving event (e.g., transaction) data associated with a plurality of events (e.g., payment transactions). First aggregation of interest data associated with a type of aggregation of interest may be received. A first key associated with each event (e.g., transaction) may be determined based on a first portion of the event (e.g., transaction) data associated with each event (e.g., transaction) and the first aggregation of interest data. A first value based at least partially on a first plurality of the first keys associated with a first subset of the plurality of payment transactions may be communicated based on a first user request. A system and computer program product are also disclosed.