Automated Testing of Code Modules Using Individualized Real-World Datasets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing automated problem detection and remediation systems face challenges in creating machine consumable IC rules that accurately detect issues across diverse device configurations, leading to potential crashes, false positives, or false negatives due to unforeseen device configurations.
Innovation Solution
A testing system that generates individualized test datasets from real-world data, including worst-case and technology-specific datasets, to validate and improve IC rules by running them against a wide range of input configurations, ensuring they function correctly before deployment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If static input datasets are used for continuous integration and unit testing, then testing process is simple and fast, but the test results may not reflect real-world device configurations leading to false positives or false negatives
Solution Approach 1:
The system performs preliminary actions by collecting and storing operational datasets from real-world device configurations before they are needed for testing. These datasets are gathered in advance and stored in a database, so when code modules need testing, realistic test data is already available, allowing both fast execution and high reliability.
Solution Approach 2:
The system creates copies of real operational datasets from production environments and uses these copies for testing code modules. Instead of using simplified static data, the system replicates actual device configurations and operational data, enabling tests to reflect real-world scenarios while maintaining testing efficiency.
2Adaptability or versatility
If IC rules are created to handle diverse device configurations, then coverage is improved, but complexity of creating and maintaining IC rules increases
Solution Approach 1:
The system enables self-service by automatically collecting operational datasets from real devices and using them to test and validate IC rules. Instead of manually creating test cases for every device configuration, the system self-generates appropriate test data from actual device operations, reducing the complexity of IC rule creation while maintaining comprehensive coverage.
Solution Approach 2:
The system applies parameter changes by dynamically selecting and applying different operational datasets with varying device configurations to test IC rules. Rather than hardcoding complex configuration handling into IC rules, the system changes the input parameters (device configurations) to test rule robustness, simplifying IC rule development while improving adaptability.
3Reliability
If more comprehensive test datasets are generated and used, then code module reliability is improved, but testing time and computational resources increase
Solution Approach 1:
The system extracts only the necessary operational datasets from the full collection of real-world data based on specific testing criteria and code module requirements. Instead of testing against all possible device configurations, the system intelligently selects and extracts relevant subsets of test data, maintaining high reliability while reducing testing time and computational overhead.
Data Source
AI summary
A testing server tests new code modules, or re-tests previously tested code modules, using individualized sets of test data. The testing server receives test datasets from an execution engine, which runs tested code modules to process operational datasets of computing devices. The testing server receives an untested code module from a user, and obtains test dataset parameters applicable to the untested code module. The testing server generates an individualized set of test datasets based on the test dataset parameters. The testing server tests the untested code module by processing each test dataset in the individualized set of test datasets to produce test results.