Streaming Aggregation Engine for Electronic Transaction Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing electronic payment processing systems face scalability issues and latency problems when handling large volumes of transactions, as they slow down due to SQL queries for real-time analysis, and replicating data to a slave database introduces delays in analysis.
Innovation Solution
Implementing a system with a metrics conversion engine, aggregation engine, and aggregation database that converts and aggregates real-time transaction data into generic metrics, allowing for flexible ad-hoc queries and user-defined functions, thereby separating analysis from transaction processing and enabling real-time analysis without impacting processing speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If SQL queries are run on the master database for real-time analysis, then analysis accuracy is improved, but transaction processing speed deteriorates
Solution Approach 1:
The system divides the database into master database (for transaction processing) and slave database (for analysis). SQL queries for analysis are segmented to run only on the slave database, preventing them from interfering with transaction processing on the master database. This spatial segmentation of database operations resolves the contradiction between analysis accuracy and processing speed.
Solution Approach 2:
The slave database acts as an intermediary between the master database and analysis queries. Instead of querying the master database directly, the system uses the slave database as a mediator that provides analysis data without impacting the master database's transaction processing performance. This intermediary approach maintains analysis accuracy while protecting transaction processing speed.
2Speed
If data is replicated to a slave database for analysis, then transaction processing speed is improved, but analysis latency increases
Solution Approach 1:
The system performs preliminary actions by continuously replicating transaction data from the master database to the slave database in near-real-time. This preliminary data preparation ensures that when analysis queries are executed on the slave database, the data is already available and synchronized, minimizing analysis latency while maintaining high transaction processing speed on the master database.
3Productivity
If the system scales to handle large transaction volumes, then productivity is improved, but system complexity increases
Solution Approach 1:
The system uses segmentation to handle large transaction volumes by dividing database operations into distinct segments: the master database handles transaction processing while the slave database handles analysis queries. This segmentation allows the system to scale horizontally by adding more slave databases for analysis without impacting the master database's transaction processing capacity, thereby improving productivity while managing system complexity through clear functional separation.
Data Source
AI summary
A new approach is proposed that contemplates systems and methods to support streaming aggregation for analysis of electronic transactions. First, a plurality of metrics to be measured/analyzed for a stream of real life events, such as processing steps of the electronic transactions are defined and converted to one or more generic metrics for aggregation. In some embodiments, the plurality of metrics flexibly include ad-hoc aggregation measures as well as various user-defined functions (UDFs), which allow a user/processor/evaluator of the electronic transactions to define and collect various types of information of the electronic transactions for analysis. Once converted, the generic metrics of the stream of real life events are aggregated by an aggregation engine in real time. The aggregation results are then saved in an aggregation database, which is queried by the user for real time analysis of the electronic transaction.