Testing Data Sanitization System for Schema-Aware Refresh
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional testing methods use example data that may not reflect real-world data, leading to obscured coding errors, security and privacy concerns with real-world data, and inefficient data generation without immediate need, causing computing inefficiencies.
Innovation Solution
A system that sanitizes and refreshes testing data by processing real-world data from a production environment, replacing sensitive information with substitute data, and providing it to a low-level testing environment on an as-needed basis, accounting for schema differences and reducing resource-intensive data generation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If real-world data is used for testing, then testing accuracy and reliability are improved, but security and privacy risks increase
Solution Approach 1:
The patent introduces an intermediary data sanitization process that transforms real-world production data into testing data by replacing sensitive information with substitute data. This intermediary step allows the system to use real-world data structures and patterns for accurate testing while eliminating security and privacy risks through systematic replacement of sensitive fields.
Solution Approach 2:
The patent creates copies of production data tables and selectively replaces sensitive information while preserving the overall data structure and relationships. This copying approach maintains testing realism by replicating production data patterns while ensuring security through controlled modification of sensitive elements.
2Object-affected harmful factors
If example data is used for testing, then security concerns are reduced, but testing reliability deteriorates
Solution Approach 1:
Instead of using fabricated example data, the patent employs real production data as an intermediary source, transforming it through systematic sanitization. This approach maintains the authenticity and complexity of real-world data patterns while ensuring security through automated replacement of sensitive information, thereby improving testing reliability compared to simple example data.
3Quantity of substance
If testing data is continuously generated, then data availability is improved, but computing efficiency deteriorates
Solution Approach 1:
The patent implements periodic data generation triggered by specific events or schedules rather than continuous generation. The system generates testing data on-demand when testing cycles are initiated, using pre-established data templates and sanitization rules, thereby reducing unnecessary computing overhead while maintaining adequate data availability for testing operations.
Solution Approach 2:
The patent performs preliminary data sanitization and template creation during system setup or off-peak periods, preparing sanitized data templates in advance. This preliminary action reduces the computational burden during actual testing operations, as the system can efficiently replicate and utilize pre-sanitized templates rather than performing full sanitization processes on-demand.
4Reliability
If production data is directly copied to testing environment, then data realism is improved, but schema compatibility issues arise
Solution Approach 1:
The patent introduces an intermediary data transformation layer that mediates between production and testing environments. This intermediary process automatically maps production data schemas to testing environment schemas, handling differences in data types, constraints, and relationships. The transformation layer preserves data realism by maintaining the essence and patterns of production data while adapting to testing environment requirements.
Solution Approach 2:
The patent systematically changes data parameters during the sanitization and transformation process, including data types, nullability constraints, and relationship definitions. These parameter changes enable schema compatibility between production and testing environments while preserving the underlying data patterns and relationships necessary for realistic testing scenarios.
Data Source
AI summary
Embodiments of the present invention provide a system for refreshing data within the testing environment by sanitizing production data. In particular, the system may process real-world data obtained in the production environment and automatically replace sensitive or confidential information with realistic substitute data, then provide the sanitized data to the systems within the low level testing environment. The system may account for differences in table schema amongst a plurality of disparate databases within the production environment and the low level environment, thereby allowing the system to provide sanitized data only to the extent that it is needed for testing purposes. Furthermore, the system selectively provides sanitized test data on an as-needed basis, removing the need to continuously generate testing data.