Dynamic Shadow Testing for Machine Learning Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for testing computer services before live deployment are inefficient, labor-intensive, and risk-prone, especially for electronic payment processing networks, as they require manual setup and lack dynamic replication of live data, leading to slow and inaccurate testing processes.
Innovation Solution
A system and method for operating dynamic shadow testing environments using machine-learning models to automatically replicate transaction data in a shadow testing environment, allowing for real-time testing with efficient resource allocation and modification of testing policies during runtime.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If manual test environment setup is used, then testing can be performed, but the process becomes labor-intensive and slow
Solution Approach 1:
The system pre-generates synthetic test data and prepares shadow testing environments before actual testing begins. Testing policies are configured in advance, and the shadow environment is populated with realistic transaction data patterns, eliminating the need for manual setup during testing operations.
Solution Approach 2:
The patent creates a shadow testing environment that copies the structure and data patterns of the production environment. Synthetic transaction data replicates live data characteristics, allowing realistic testing without manual environment configuration or access to actual production data.
2Quantity of substance
If historic data duplication is used for testing, then data availability is improved, but the flow and timing replication accuracy deteriorates
Solution Approach 1:
The system transforms static historic data into dynamic synthetic data that replicates temporal patterns, flow characteristics, and timing behaviors of live transactions. Machine learning models adjust data parameters to match production environment patterns, achieving both data availability and temporal accuracy.
Solution Approach 2:
The patent replaces manual or simple automated data copying mechanisms with machine learning-based synthetic data generation. The ML models learn complex temporal and behavioral patterns from historic data and generate new synthetic transactions that accurately replicate production flow and timing characteristics.
3Measurement precision
If live data testing is performed, then testing accuracy is improved, but the risk of data overwriting and service actions increases
Solution Approach 1:
The system introduces a shadow testing environment as an intermediary layer between testers and production data. This shadow environment contains synthetic data that mirrors production characteristics, allowing accurate testing while completely isolating the production system from testing operations and eliminating data overwriting risks.
Solution Approach 2:
Instead of testing directly on live data, the patent creates accurate copies of the production environment structure and data patterns in a shadow testing environment. These synthetic copies enable realistic testing scenarios while maintaining complete separation from actual production data, preventing any harmful side effects.
4Device complexity
If static test environments are used, then resource allocation is simplified, but the ability to perform iterative testing deteriorates
Solution Approach 1:
The system transitions from static test environments to dynamic shadow testing environments that can be automatically created, modified, and destroyed on demand. Testing policies can be changed at runtime, and the shadow environment adapts to new testing requirements without manual reconfiguration, enabling rapid iterative testing while maintaining automated resource management.
Data Source
AI summary
Described are a method, system, and computer program product for operating dynamic shadow testing environments for machine-learning models. The method includes generating a shadow testing environment operating at least two transaction services. The method also includes receiving a plurality of transaction authorization requests. The method further includes determining a first percentage associated with a first testing policy of the first transaction service and a second percentage associated with a second testing policy of the second transaction service. The method further includes replicating in the shadow testing environment, in real-time with processing the payment transactions, a first portion of the plurality of transaction authorization requests and a second portion of the plurality of transaction authorization requests. The method further includes testing the first transaction service using the first set of replicated transaction data and the second transaction service using the second set of replicated transaction data.