Database Data Reduction for Testing Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large companies face high costs and storage challenges due to the need for extensive physical space and resources to manage and replicate multi-terabyte databases for testing and maintenance, as traditional test databases do not reasonably approximate the characteristics of large production databases.
Innovation Solution
A system that gathers statistics from a production database to create a smaller test database instance, focusing on data skew and key locations, allowing for efficient replication and maintenance while maintaining performance characteristics, using a purge program to remove unnecessary data and an optimizer to preserve data distribution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a test database is created as a copy of the production database, then testing functionality is improved, but physical storage costs and space requirements increase significantly
Solution Approach 1:
The patent extracts only the essential characteristics and data distributions from the production database to create a reduced test database. By identifying and retaining only the critical data samples that maintain performance characteristics, the system creates a smaller test database that does not require full replication of the production database, thus reducing physical storage requirements while maintaining testing reliability.
Solution Approach 2:
The patent changes the scale and composition parameters of the test database by using statistical sampling and data reduction techniques. Instead of creating a full copy, the system transforms the production database into a reduced version that maintains key statistical properties and performance characteristics, thereby reducing storage volume while preserving testing functionality.
2Adaptability or versatility
If multiple copies of the production database are created for testing different scenarios, then testing coverage is improved, but device complexity and resource requirements increase
Solution Approach 1:
The patent creates a single reduced test database that can serve multiple testing purposes and scenarios. By designing the reduced database to maintain essential performance characteristics and data distributions, the same test database can be used for various testing scenarios including performance testing, upgrade testing, and functional testing, eliminating the need for multiple specialized database copies and reducing overall system complexity.
3Volume of stationary object
If data is reduced from the production database, then storage costs are reduced, but data representation accuracy may deteriorate
Solution Approach 1:
The patent applies different data retention strategies to different parts of the database based on their importance. By identifying critical data samples, key distributions, and essential performance characteristics, the system retains data selectively in specific locations and ranges while reducing or removing less critical data. This localized approach maintains data representation accuracy for testing while significantly reducing overall storage requirements.
Data Source
AI summary
A reasonably-sized testing database instance can be efficiently replicated and maintained for a very large production database while retaining the characteristics and cross-sectional data. The performance characteristics are maintained in order to provide for proper testing of the production database for various application programs. Statistics on the type of data distribution for the customer data are obtained, allowing for parameters to be determined which can be used to store data only near the endpoints of the distribution (and/or at other key locations). In this way, a substantial amount of data skew is retained in a much smaller instance of the production database, allowing for easier performance testing, upgrade testing, etc.