Machine Learning Program Verification Using Satisfiability Solvers

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Evaluating the validity of a prediction model with a large number of decision trees is challenging due to time and cost constraints, especially when the input value space is vast, leading to potential system failures if untested input values are not evaluated.

Innovation Solution

A machine learning program verification apparatus that creates decision tree logical expressions, combines them with verification property and objective variable calculation expressions, and uses satisfiability determiners to assess whether the model satisfies specific properties, identifying violating input values and output values, thereby enabling comprehensive evaluation.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If a large number of decision trees are created to achieve practical generalization performance, then the prediction model's accuracy is improved, but the time and cost required for evaluation increases significantly

Engineering Contradiction:

Improvegeneralization performanceVSAvoidevaluation time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent replaces the mechanical approach of manually testing each input value with an automated formal verification system that uses satisfiability solvers and symbolic execution to mathematically prove or disprove model correctness, dramatically reducing evaluation time while maintaining comprehensive coverage

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The patent performs preliminary symbolic execution and path analysis before actual verification, pre-computing decision tree paths and conditions to streamline the subsequent formal verification process, reducing the overall evaluation time for large ensembles

Inventive Principle:

Principle #10Preliminary action

2Reliability

If all input values are tested to ensure comprehensive evaluation, then the reliability of the prediction model is improved, but the complexity and cost of the evaluation process increases

Engineering Contradiction:

Improvecomprehensive evaluationVSAvoidevaluation complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent extracts and analyzes individual decision tree paths separately using symbolic execution, then combines the results through formal verification. This decomposition approach manages complexity by handling one path at a time rather than attempting to analyze all input values simultaneously

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent introduces formal verification properties and satisfiability solvers as intermediaries between the prediction model and the evaluation process. These intermediaries translate complex verification tasks into standardized logical formulas that can be systematically solved, reducing overall evaluation complexity

Inventive Principle:

Principle #24Intermediary (Mediator)

3Ease of operation

If visual confirmation of decision tree logic is performed, then the understandability of the model is improved, but the evaluation coverage remains limited when many trees are present

Engineering Contradiction:

Improvevisual confirmationVSAvoidevaluation coverage

Core Design Contradiction:

Ease of operationVSMeasurement precision

Solution Approach 1:

The patent replaces manual visual confirmation with automated formal verification that systematically checks all decision tree paths against specified properties. This substitution maintains the benefit of thorough evaluation while eliminating the limitation of human capacity to visually inspect only a few trees

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Data Source

PatentUS11481692B2Machine learning program verification apparatus and machine learning program verification method

Publication Date: 2022.10.25 HITACHI LTD

AI summary

A validity of a prediction model can be evaluated comprehensively. A machine learning program verification apparatus 100 includes a calculation device 104. The calculation device 104 obtains a decision tree logical expression by logically combining path logical expressions indicating decision tree paths indecision trees for a program created by machine learning, creates a combined logical expression by logically combining a verification property logical expression and an objective variable calculation logical expression with the decision tree logical expression, performs satisfiability determination by inputting the combined logical expression to a satisfiability determiner, and when a result of the determination indicates satisfaction, obtains, from a satisfaction solution of the satisfiability determination, a violation input value that is a value of an explanatory variable that violates a verification property and a violation output value that is a value of an objective variable.