LLM-Based Test Procedure Generation from Domain Documents
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current testing processes for software and electronics components rely heavily on manual generation of test cases, which is time-consuming and often results in non-exhaustive test cases due to insufficient domain knowledge, and automation fails to provide efficient solutions.
Innovation Solution
A method and system using Large Language Models (LLMs) to determine test procedures by receiving domain-based documents, extracting domain-specific keywords, generating contextual data and knowledge datasets, and creating test cases and procedures based on these inputs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual generation of test cases is used, then domain knowledge can be applied to create test cases, but it consumes significant time and may still result in non-exhaustive test cases
Solution Approach 1:
The patent segments the test case generation process into multiple independent LLM components: one LLM extracts domain knowledge from documents, another generates test cases based on extracted knowledge, and a third validates test case completeness. This segmentation allows each component to specialize in one task, improving overall reliability while maintaining efficiency through parallel processing capabilities.
Solution Approach 2:
The patent introduces an intermediary knowledge extraction layer between domain documents and test case generation. This intermediary LLM processes domain documents to extract structured knowledge, which then serves as input for the test case generation LLM. This intermediary ensures that domain knowledge is systematically captured and transferred, improving test case completeness without requiring manual intervention for each test case.
2Productivity
If automation in testing is used, then time consumption is reduced, but it fails to provide exhaustive test cases due to lack of domain knowledge
Solution Approach 1:
The patent performs preliminary action by having one LLM extract and structure domain knowledge from documents before the test case generation process begins. This pre-processing of domain knowledge ensures that the test case generation LLM has comprehensive domain context available, enabling it to generate exhaustive test cases automatically without missing critical domain-specific scenarios.
Solution Approach 2:
The patent replaces the mechanical system of manual test case creation with an intelligent system using LLMs. The LLM-based system automatically processes domain documents, extracts knowledge, and generates test cases, substituting human manual effort with automated AI processing that maintains both efficiency and domain knowledge integration.
3Measurement precision
If multiple LLMs are used to process domain knowledge, then accuracy of test procedures is improved, but system complexity increases
Solution Approach 1:
The patent segments the LLM system into specialized components with distinct responsibilities: knowledge extraction LLM, test case generation LLM, and validation LLM. Each segment focuses on a specific task, improving accuracy through specialization while managing complexity by keeping each component relatively simple and well-defined.
Solution Approach 2:
The patent employs universal LLM components that can perform multiple functions within their designated roles. Each LLM is trained to handle various aspects of its specific task (e.g., the knowledge extraction LLM can extract different types of domain knowledge from various document formats), providing multi-functionality that reduces the need for numerous specialized models and thereby manages system complexity.
Data Source
AI summary
A method and system of determining test procedures using large language models (LLMs) is disclosed. A processor receives a plurality of domain-based documents corresponding to a test product to be tested. One or more user requirements are determined corresponding to at least one feature of the test product from one of the plurality of domain-based documents. A plurality of domain-specific keywords is determined from the one or more user requirements. A contextual data is determined by extracting a portion of a text data from the plurality of domain-based documents. A knowledge dataset is determined by prompting a second LLM based on a second prompt and the contextual data. One or more test procedures for one or more test cases for testing the at least one feature of the test product by prompting a third LLM based on a third prompt and the knowledge dataset.