AI Test Plan Validation Using Similarity Thresholds
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Generative AI in avionics system testing is susceptible to hallucinations, leading to the generation of incorrect and/or incomplete test plans, posing a risk to aircraft safety.
Innovation Solution
A method and apparatus utilizing a state machine and neural network to compare new test plan requirements with pre-existing plans, determining similarities exceeding threshold levels, and generating a new test plan using generative AI while ensuring sufficient similarity and completeness, involving pre-defined templates and environment selection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If generative AI is used to automatically generate test plans, then productivity is improved, but reliability deteriorates due to hallucinations producing incorrect or incomplete test plans
Solution Approach 1:
The patent introduces an intermediary validation system that acts as a mediator between the generative AI and the final test plan output. This validation system checks generated test plans against pre-existing test plans and requirements, identifying hallucinations and errors before the test plan is finalized. The intermediary layer preserves the productivity benefits of AI generation while mitigating reliability issues through automated validation.
Solution Approach 2:
The patent implements a feedback mechanism where the generated test plan is compared against pre-existing test plans and requirements, and the results are fed back to validate correctness. This feedback loop allows the system to identify and correct hallucinations by checking whether generated content aligns with established patterns and requirements, thereby improving reliability without sacrificing AI-generated productivity.
2Reliability
If similarity thresholds are applied to validate test plans, then reliability is improved, but device complexity increases due to additional validation components
Solution Approach 1:
The patent uses parameter changes in the form of similarity thresholds to validate test plans. By adjusting the similarity threshold parameter, the system can control the strictness of validation without changing the overall validation architecture. This allows reliability to be improved through parameter tuning rather than adding complex validation logic, thereby limiting the increase in device complexity.
Solution Approach 2:
The validation system is segmented into distinct components: similarity determination module, threshold comparison module, and validation decision module. This segmentation allows each component to perform a specific function, making the overall system easier to manage and understand. The modular structure reduces perceived complexity while maintaining high reliability through systematic validation.
3Manufacturing precision
If pre-existing test plans are compared with new test plan requirements, then manufacturing precision is improved, but loss of time increases due to additional comparison steps
Solution Approach 1:
The patent applies preliminary action by pre-processing and indexing pre-existing test plans before validation. This preliminary preparation allows the similarity determination to occur more efficiently during the actual validation process. By doing the heavy lifting of data preparation in advance, the system achieves high manufacturing precision in requirement alignment without excessive time loss during the critical validation phase.
Data Source
AI summary
Techniques are generating a test plan, based on test requirements, using artificial intelligence (AI) whilst avoiding hallucinations. Hallucinations in the AI generated test plan may be reduced by utilizing generative AI and providing the generative AI with pre-defined custom templates, test plan guidelines, and/or information for selecting a test environment based on test type. Additionally or alternatively, a similarity between the test plan and a similar test plan may be determined to avoid hallucinations.