Cloud ML Model Testing Across Tenant Datasets for Fraud Validation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Financial institutions face challenges in creating robust Machine Learning (ML) models due to low fraud counts, lack of data for Hyper Parameter Optimization (HPO), and hesitation to deploy models due to concerns over unpredicted data patterns, leading to increased losses.
Innovation Solution
A computerized-method and system for testing and training classification ML models in a cloud-based environment, utilizing APIs to evaluate models across similar datasets of other tenants, perform Hyper Parameter Optimization, and calculate performance scores to ensure robustness before deployment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If financial institutions use most of the fraud transaction data for training the ML model, then the training data utilization is maximized, but the testing data becomes insufficient leading to poor model validation
Solution Approach 1:
The patent creates synthetic fraud transaction data that replicates the characteristics of real fraud patterns. This synthetic data is then used to augment the training set, allowing the model to be trained on a larger diverse set of fraud examples without depleting the actual limited fraud transaction data for testing purposes.
Solution Approach 2:
The patent divides the fraud detection task into separate training and validation phases using synthetic data for training and reserved real data for validation. This segmentation allows independent optimization of training quality while preserving sufficient real fraud data for reliable model testing and validation.
2Productivity
If financial institutions use default set of parameters for training the ML model, then the training process is simplified and faster, but the model performance becomes suboptimal due to lack of Hyper Parameter Optimization
Solution Approach 1:
The patent implements Hyper Parameter Optimization by systematically varying key parameters such as the number of trees, maximum depth, and learning rate in the Random Forest model. This allows the model to achieve optimal performance on the synthetic training data while maintaining the ability to generalize to real-world fraud patterns.
Solution Approach 2:
The patent employs cross-validation and performance metrics to evaluate different hyper parameter configurations. The validation process provides feedback that guides the selection of optimal hyper parameters, ensuring the model achieves high accuracy and robustness before deployment.
3Productivity
If financial institutions deploy ML models without comprehensive testing, then the deployment process is accelerated, but the risk of undetected fraud patterns and losses increases
Solution Approach 1:
The patent performs comprehensive model validation and testing using synthetic data before actual deployment. This preliminary action includes evaluating the model's performance on diverse fraud patterns, checking for biases, and ensuring robustness against various attack vectors, thereby reducing the risk of undetected fraud patterns in production.
Data Source
AI summary
A computerized-method for testing a classification ML model of a tenant of a service provider, in a cloud-based environment. The computerized-method includes: (i) receiving an object of a classification ML model for testing from the tenant; (ii) executing an API with the received object of the classification ML model; (iii) identifying one or more tenants of the service provider based on an activity type and preconfigured characteristics by the executed API; (iv) performing an evaluation of the object of the classification ML model by operating the API on each retrieved dataset of the one or more tenants of the service provider to evaluate the object of the classification ML model and store score-results; and (v) calculating an average of the stored score-results to yield a performance-score of the classification ML model. When the performance-score is above a predefined performance-score deploying the classification ML model in a system of the tenant.