Automated Software Performance Testing via Test Data Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Manual creation of tests, test data, and analysis of software performance is time-consuming, cumbersome, and expensive, especially in complex software landscapes with lengthy lifecycles and frequent updates.
Innovation Solution
A system that generates corpuses of test data of varying scope by extrapolating from small original test samples using repetition techniques for numerical data and machine learning models for complex data, allowing for automated measurement of performance parameters like runtime, memory consumption, and bandwidth.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual creation of tests and test data is performed, then measurement precision is improved, but loss of time and productivity deteriorate
Solution Approach 1:
The system performs preliminary actions by automatically generating test data corpuses from small original test samples before actual performance testing. This pre-generation of test data eliminates the time-consuming manual creation process while ensuring the test data is ready and appropriate for accurate performance measurement of software functions throughout their lifecycle.
Solution Approach 2:
The system creates copies by generating large corpuses of test data from small original test samples. For numerical data, repetition techniques create copies; for complex data including text and hierarchical structures, machine learning models generate copies that maintain the essential characteristics of the original samples while providing sufficient volume for comprehensive testing.
2Measurement precision
If manual creation of tests and analysis is performed, then measurement precision is improved, but productivity deteriorates
Solution Approach 1:
The system performs self-service by automatically generating test data corpuses and analyzing performance results without requiring manual intervention. The automated analysis of performance parameters and identification of non-linear behavior patterns enables the system to serve itself in completing the entire testing workflow, thereby maintaining measurement precision while dramatically improving productivity.
Solution Approach 2:
The system replaces the mechanical manual process of test creation and analysis with automated computational processes. Machine learning models and repetition techniques substitute for manual test data generation, while automated analysis tools replace manual performance assessment, thereby maintaining accuracy while enhancing productivity.
3Reliability
If manual creation of tests is performed, then reliability is improved, but loss of time and productivity worsen
Solution Approach 1:
The system implements feedback mechanisms by automatically analyzing performance parameters and identifying non-linear behavior patterns. This feedback loop ensures that test results are reliably interpreted and that the generated test data corpuses are appropriate for their intended testing purposes, maintaining reliability while reducing the time required for manual analysis.
4Ease of manufacture
If small volume test samples are used, then ease of manufacture is improved, but measurement precision deteriorates
Solution Approach 1:
The system performs preliminary action by automatically expanding small original test samples into large corpuses of test data before the actual performance testing begins. This pre-expansion ensures that sufficient test data volume is available for accurate measurement while the original small sample size maintains ease of initial test creation.
Solution Approach 2:
The system creates multiple copies of the original small test samples to generate large corpuses of test data. Through repetition techniques for numerical data and machine learning models for complex data, the system produces numerous copies that provide the volume needed for precise performance measurement while starting from easily manageable small samples.
Data Source
AI summary
Embodiments facilitate testing of software functions, by elaborating upon a small volume test sample originally provided by a developer. Where the original test sample is determined to be numerical, embodiments perform repetition techniques to generate corpuses of test data of varying scope representing operation over different timescales. For original test samples of greater complexity (e.g., including text and/or hierarchical output states of the software function), embodiments select a relevant Machine Learning (ML) model and apply that ML model to generate the corpuses of test data. These are then available for loading to measure one or more performance parameters (e.g., runtime, memory consumption; processing consumption; bandwidth) of the software function. Broadly reported non-linear behavior of performance parameter(s) may reveal lack of suitability of the generated data corpuses for testing. More specific non-linear behavior may indicate failure of the software function to pass certain tests.