Metamorphic Test Input Generation for Transitive Consistency Checks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current metamorphic testing solutions lack a fully automatic approach, making it difficult to effectively test systems where calculating expected outputs is impractical, particularly in contexts like machine learning models and context-sensitive data.
Innovation Solution
A system and method that utilizes the system under test to generate metamorphic testing-ready datasets, enabling partial or full automation of the testing process by identifying relationships between inputs and outputs, and testing transitive consistency using a data store of potential inputs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual metamorphic testing is performed, then testing thoroughness is improved, but labor cost and time consumption increase
Solution Approach 1:
The system performs preliminary actions by automatically generating test inputs and expected outputs before actual testing occurs. The metamorphic relationship generator pre-establishes the testing framework, input generation rules, and expected output criteria, eliminating the need for manual test case preparation during execution.
Solution Approach 2:
The testing system serves itself by automatically generating test inputs, computing expected outputs, and executing tests without human intervention. The system uses its own resources (processors, memory, data stores) to perform all testing operations, making the manual labor unnecessary while maintaining thoroughness.
2Measurement precision
If manual input generation is performed, then input quality is improved, but productivity decreases
Solution Approach 1:
The system performs preliminary actions by automatically generating test inputs and expected outputs before actual testing occurs. The metamorphic relationship generator pre-establishes the testing framework, input generation rules, and expected output criteria, eliminating the need for manual test case preparation during execution.
Solution Approach 2:
The patent replaces the mechanical manual process of input generation with an automated computational system. The system uses algorithms and data structures to generate inputs programmatically, substituting human manual work with machine-based automation that maintains quality while dramatically increasing productivity.
3Measurement precision
If expected output calculation is performed manually, then testing accuracy is improved, but device complexity increases
Solution Approach 1:
The system extracts the complex expected output calculation process into a separate, dedicated module (the output generator). This module is specifically designed to compute expected outputs based on metamorphic relationships, isolating the complexity from the main testing system and making it a reusable, standardized component.
Solution Approach 2:
The patent replaces the mechanical manual process of input generation with an automated computational system. The system uses algorithms and data structures to generate inputs programmatically, substituting human manual work with machine-based automation that maintains quality while dramatically increasing productivity.
Data Source
AI summary
Computer hardware and/or software that performs the following operations: (i) identifying a test input for a system under test, the system under test configured to determine a strength of a relationship between received inputs; (ii) determining, from a data store of potential inputs, a set of inputs for which the strength of the relationship to the identified test input according to the system under test meets a first threshold; and (iii) utilizing the determined set of inputs to test a transitive consistency of an output of the system under test.