ML Prediction Model Verification Using Approximation Ranges
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing machine learning model verification methods face challenges in achieving accurate and efficient validation due to exponentially increasing calculation amounts with the number of decision trees, making it difficult to verify prediction models with appropriate accuracy.
Innovation Solution
A software test apparatus and method that simplifies logical expressions by analyzing approximation ranges based on an approximation threshold value, generates inspection expressions, and searches for counterexamples to evaluate the prediction model's validity efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the prediction model includes more decision trees to improve accuracy, then the verification accuracy is improved, but the calculation amount increases explosively
Solution Approach 1:
The patent segments the verification process into two stages: first converting the prediction model to a logical expression and extracting approximation ranges, then performing satisfiability determination only within those ranges. This segmentation allows verification of large-scale models with many decision trees without explosive calculation increase, as the approximation range extraction is performed once and the satisfiability determination is constrained to reduced search spaces.
Solution Approach 2:
The patent performs preliminary action by converting the prediction model to a logical expression and extracting approximation ranges before performing satisfiability determination. This preliminary processing identifies and stores the relevant input ranges that could potentially violate constraints, so that the subsequent satisfiability determination only needs to search within those pre-identified ranges rather than the entire input space, significantly reducing calculation amount while maintaining verification accuracy.
2Measurement precision
If the prediction model includes more decision trees to improve accuracy, then the verification accuracy is improved, but the device complexity increases
Solution Approach 1:
The patent extracts the approximation ranges from the logical expression representation of the prediction model, separating the complex model structure from the verification process. By converting the model to a logical expression and extracting only the relevant approximation ranges, the method handles complex models with many decision trees without proportionally increasing verification complexity, as the extraction process captures essential constraints without replicating full model complexity.
3Measurement precision
If the approximation threshold value is reduced to improve accuracy, then the detection error is reduced, but the calculation amount increases
Solution Approach 1:
The patent applies partial action by using approximation ranges extracted from the logical expression to constrain the satisfiability determination search space. Even with a relatively coarse approximation threshold, the verification is performed only within the extracted approximation ranges rather than the entire input space, achieving acceptable detection accuracy without proportional increases in calculation amount. This partial verification approach balances accuracy and computational feasibility.
Data Source
AI summary
To make validity of a prediction model created by machine learning be able to be verified with appropriate accuracy and efficiency. A software test apparatus includes a storage device configured to store a prediction model, and an arithmetic device. The arithmetic device is configured to accept inputs of a precondition, a constraint condition, and an approximation threshold value, convert the prediction model into a logical expression, analyze an approximation range based on the approximation threshold value with respect to the logical expression to simplify the logical expression, generate an inspection expression by combining the simplified logical expression with the precondition and negation of the constraint condition, search for, as a counterexample, a value satisfying the inspection expression, input the value to the prediction model to evaluate inspection accuracy when the counterexample exists, and output a result of the evaluation.