Production Data Source Replay for Realistic Performance Testing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing software performance tests are heavily dependent on test data structure and load, which often differ from actual production environments, leading to inaccurate performance evaluations due to missing or unrealistic test data and varying hardware configurations.
Innovation Solution
A method to conduct performance tests on customer systems using a test version of the software application, recording how data sources are accessed in production, and comparing performance between current and test versions to adjust code changes for accurate results.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If automated performance tests are conducted using internal test systems, then performance testing can be performed automatically, but the test results are strongly dependent on test data structure and load which differ from production environments
Solution Approach 1:
The patent creates a copy of the production system's data access patterns by recording how data sources are accessed in the production environment and replaying those same access patterns in the test environment. This copying approach ensures that test data structure and load closely match production conditions, thereby improving measurement precision while maintaining automated testing capabilities
Solution Approach 2:
The system performs preliminary recording of data access patterns from the production environment before conducting performance tests. By capturing and storing the actual data access behaviors, query types, and frequencies in advance, the test system can later replay these patterns with high fidelity, ensuring accurate performance measurements that reflect real production conditions
2Productivity
If test data is created to pass developer's test checks, then testing can be completed, but the quality of performance test data is often bad leading to frequent performance issues
Solution Approach 1:
The system establishes a feedback loop where performance test results from the test environment are compared against actual production performance data. When discrepancies are detected, the test data generation process is adjusted and refined based on this feedback, continuously improving test data quality and reducing performance issues in production
Solution Approach 2:
The system automatically generates high-quality test data by extracting and replaying actual data access patterns from production environments, eliminating the need for manual test data creation. This self-service approach ensures test data naturally reflects real production conditions without requiring expert intervention, thereby improving both productivity and reliability
3Adaptability or versatility
If customer's hardware configuration is used for testing, then realistic production conditions are reflected, but hardware configuration might be different or less efficient than testing hardware
Solution Approach 1:
The system dynamically adjusts test parameters such as data set sizes, query complexity, and concurrency levels to match the specific hardware configuration of the customer's production environment. By adapting these parameters rather than using fixed test configurations, the system achieves accurate performance measurements across diverse hardware platforms without sacrificing test realism
Data Source
AI summary
To conduct realistic performance analysis for planned software changes, code changes are obtained for an application, including a change to a data source. Then a test version of the application and test information are provided to a separate production system running a current version of the application. The test version includes the code changes and the test information includes a request to test the data source. Later, test results are received from the production system. The test results compare performance of the data source in the current version of the software application to performance of the data source in the test version of the software application. The test results are obtained by supervising the data source, recording how the data source was accessed during operation of the production system, and testing performance in test version in a same manner as it was accessed during the operation.