Synthesizing Test Data via Topic Modeling for AI Model Verification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional software testing methods are inadequate for Artificial Intelligence/Machine Learning (AI/ML) models due to their lack of structured code, and existing approaches are impractical for neural networks, necessitating a different testing strategy to validate and improve AI/ML Engines, especially in high-dimensional data environments like insurance claims processing.
Innovation Solution
The implementation of novel 'black-box' differential testing techniques using a pre-processed topic model, such as Latent Dirichlet Allocation (LDA) Topic Model, to guide the generation of synthesized test data, reducing the search space and improving the efficiency of testing AI/ML models by focusing on specific input modifications within classified data subsets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional white-box testing methods are used on AI/ML models, then code verification is improved, but the approach becomes impractical because AI/ML models lack precisely formulated software code
Solution Approach 1:
The patent inverts the traditional testing approach by moving from white-box testing (requiring visible code) to black-box differential testing (comparing inputs and outputs without seeing internal code). Instead of verifying code logic directly, the system verifies model behavior by comparing differential test case pairs, making testing practical for neural networks while maintaining reliability through systematic output comparison.
2Reliability
If the entire input space is searched for differential test cases, then test coverage is improved, but computing time and resources increase significantly
Solution Approach 1:
The patent segments the vast input space by introducing topic models that categorize and structure the search space. Instead of searching all possible inputs uniformly, the system divides the search into topic-based segments, allowing focused exploration of relevant regions. This segmentation maintains comprehensive coverage within each topic while dramatically reducing overall computing time by avoiding irrelevant input regions.
Solution Approach 2:
The patent performs preliminary action by pre-processing and organizing the input space using topic models before generating differential test cases. The topic models are trained beforehand to understand the structure and relationships in the data, enabling efficient navigation and selection of test inputs. This preliminary structuring avoids the need to search the entire input space during actual test generation.
3Ease of manufacture
If random test case generation is used, then simplicity is improved, but test quality and relevance to actual data distributions deteriorate
Solution Approach 1:
The patent introduces topic models as an intermediary between simple random generation and complex data-driven test generation. The topic models serve as a mediator that captures the underlying structure and distributions of actual data, then guides test case generation to reflect realistic patterns. This intermediary approach maintains the simplicity of automated generation while significantly improving test quality by ensuring relevance to actual data distributions.
Data Source
AI summary
Systems and methods for generating a dataset of synthesized data items from a dataset of original data items are disclosed herein. Some embodiments include (i) selecting an original data item from the dataset of original data items, where each original data item (a) comprises a combination of first-type codes and second-type codes, and (b) is associated with a topic in a topic model; and (ii) generating a synthesized data item based on the original data item and the topic associated with the original data item, where the synthesized data item comprises a combination of first-type codes and second-type codes that differs from the combination of first-type codes and second-type codes in the original data item by one first-type code or one second-type code.