Transaction Data Join Optimization via Candidate Row Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Prepaid card issuers face challenges in analyzing large amounts of transaction data from multiple databases to identify regulatory violations and malicious activities, as traditional join operations are inefficient due to the vast amount of uncompressed data, making it difficult for human analysts to detect patterns or inconsistencies.
Innovation Solution
A system that retrieves and reorganizes raw transaction data, uses filtering techniques to reduce processor load during join operations, and applies rules to analyze the data for regulatory compliance, generating alerts for fraud analysts through an interactive user interface, while employing machine learning to improve rule functionality over time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional join operations are used to combine data from multiple databases, then complete transaction data can be analyzed, but the processor load becomes too high and the operation becomes inefficient due to the vast amount of uncompressed data
Solution Approach 1:
The patent segments the large-scale join operation into multiple phases: first identifying candidate rows using filtering techniques, then performing join operations only on these reduced subsets. This segmentation divides the computationally intensive task into manageable chunks, reducing peak processor load while maintaining complete data analysis capability.
Solution Approach 2:
The patent applies filtering techniques as a preliminary action before executing join operations. By pre-identifying and selecting only the relevant candidate rows that meet specific criteria, the system reduces the data volume subjected to expensive join operations, thereby improving overall productivity without sacrificing analytical completeness.
2Reliability
If data from multiple databases is combined into a single database for analysis, then regulatory violations can be detected, but the join operations become difficult to run due to the large amount of uncompressed data
Solution Approach 1:
The patent extracts only the necessary candidate rows from the full databases using filtering techniques, rather than combining and processing all data. This extraction approach maintains the ability to detect regulatory violations by focusing on relevant data subsets, while making join operations feasible by dramatically reducing the data volume involved.
Solution Approach 2:
The patent segments the data combination process into filtering and joining phases, where filtering identifies candidate rows that are then joined. This segmentation makes the overall process manageable by breaking down the infeasible task of joining all data into the feasible task of joining filtered subsets.
3Measurement precision
If human fraud analysts manually analyze transaction data to identify patterns, then regulatory compliance can be assessed, but the time and effort required becomes prohibitively large due to the volume of uncompressed data
Solution Approach 1:
The patent introduces an automated filtering and candidate identification system as an intermediary between the raw data and human fraud analysts. This intermediary pre-processes the data by applying filtering techniques to identify candidate rows, thereby preserving the analytical precision that analysts need while dramatically reducing the time they must spend on manual data review.
Solution Approach 2:
The patent performs preliminary filtering and candidate identification automatically before presenting data to human analysts. This preliminary action maintains detection accuracy by ensuring relevant candidates are identified, while reducing the time loss by eliminating the need for analysts to manually scan through vast amounts of uncompressed data.
Data Source
AI summary
Various systems and methods are provided that retrieve raw data from issuers, reorganize the raw data, analyze the reorganized data to determine whether the risky or malicious activity is occurring, and generate alerts to notify users of possible malicious activity. For example, the raw data is included in a plurality of tables. The system joins one or more tables to reorganize the data using several filtering techniques to reduce the processor load required to perform the join operation. Once the data is reorganized, the system executes one or more rules to analyze the reorganized data. Each rule is associated with a malicious activity. If any of the rules indicate that malicious activity is occurring, the system generates an alert for display to a user in an interactive user interface.