Real-Time Transaction Aggregation with On-Demand Value Calculation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data aggregation techniques face challenges in efficiently and accurately processing high volumes of transaction data in real-time due to state explosion and inefficient use of computing and network resources, leading to high latency and limited flexibility.
Innovation Solution
A method and system for data aggregation that involves determining keys based on transaction data and aggregation of interest, storing data in a map structure, and calculating values on demand, reducing the need for proactive state calculation and minimizing network data transfer.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If proactive state storage for all variables is implemented, then data aggregation completeness is improved, but computing resource efficiency deteriorates due to state explosion
Solution Approach 1:
The system proactively calculates and stores aggregation values for predetermined time periods (e.g., hourly, daily aggregations) in advance. When a user requests data, the system retrieves these pre-calculated values and only computes the remaining portion, avoiding the need to store all possible aggregation states while still providing fast response for common time periods.
Solution Approach 2:
The patent divides the aggregation time period into predetermined intervals (e.g., full hours, full days) and remaining portions. The system stores aggregation values only for these segmented predetermined intervals, not for all possible time ranges. This segmentation allows the system to maintain completeness for standard periods while avoiding state explosion for custom or arbitrary time ranges.
2Measurement precision
If all incoming raw data is stored and searched upon request, then aggregation accuracy is improved, but processing latency increases
Solution Approach 1:
The system performs preliminary calculations of aggregation values for predetermined time periods and stores them in advance. When a user request arrives, the system retrieves these pre-computed values immediately, avoiding the need to search through all raw transaction data. This eliminates the time-consuming scanning process while maintaining accurate aggregation results for the requested time period.
Solution Approach 2:
The patent extracts and stores only the aggregation values for predetermined time periods from the raw data, separating this pre-computed information from the full raw dataset. When a request is made, the system extracts the relevant pre-computed portion rather than searching through all raw data, significantly reducing processing time while maintaining accuracy.
3Adaptability or versatility
If all relevant raw data portions are transmitted to user device, then data flexibility is improved, but network resource efficiency deteriorates
Solution Approach 1:
The system extracts and transmits only the specific aggregation values that are relevant to the user's request, rather than sending all raw data portions. The user device receives only the computed aggregation results needed for its specific purpose, eliminating the transmission of voluminous unnecessary data while maintaining the flexibility for the user to perform its required calculations.
Solution Approach 2:
The patent introduces an intermediary computation step where the service provider system calculates aggregation values from raw data and transmits only these computed results to the user device. This intermediary processing prevents the direct transmission of large volumes of raw data, reducing network resource usage while still providing the user device with the flexibility to perform its specific aggregation calculations.
4Adaptability or versatility
If user device performs aggregation calculations, then computing centralization is reduced, but user device resource efficiency deteriorates
Solution Approach 1:
The service provider system performs the computationally intensive aggregation calculations in advance for predetermined time periods and stores these results. When a user requests data, the system retrieves and transmits these pre-computed values, eliminating the need for the user device to perform heavy computation. This maintains computing distribution architecture while significantly improving user device resource efficiency.
Data Source
AI summary
Provided is a method for aggregating data from real-time events (e.g., payment transactions). The method may include receiving event (e.g., transaction) data associated with a plurality of events (e.g., payment transactions). First aggregation of interest data associated with a type of aggregation of interest may be received. A first key associated with each event (e.g., transaction) may be determined based on a first portion of the event (e.g., transaction) data associated with each event (e.g., transaction) and the first aggregation of interest data. A first value based at least partially on a first plurality of the first keys associated with a first subset of the plurality of payment transactions may be communicated based on a first user request. A system and computer program product are also disclosed.