Trusted Machine Learning Model Verification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning systems, particularly in life-critical and mission-critical applications, often produce unreasonable or erroneous outputs due to complexity, leading to potential safety issues without effective high-level correctness guarantees.
Innovation Solution
The development of machine learning methods and systems that use a trusted model to ensure reliability by determining trust-related constraints, modifying models, data, or reward functions through repair algorithms, and re-training until the model satisfies these constraints, providing a trusted model for operation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If machine learning algorithms are used in critical applications, then the system can perform complex learning tasks, but the system produces unreasonable or erroneous outputs due to complexity
Solution Approach 1:
The patent introduces a formal verification module as an intermediary between the machine learning model and the critical application. This module verifies model outputs against predefined constraints and specifications before execution, acting as a mediator that filters erroneous outputs while preserving the adaptive capabilities of the ML system.
Solution Approach 2:
The patent performs formal verification and validation of machine learning models before deployment in critical applications. By conducting preliminary checks on model correctness, constraint satisfaction, and expected behavior, the system prevents erroneous outputs from reaching the application layer.
2Reliability
If a bounding envelope or wrapper is used to check for violations dynamically, then assurance for complex learning systems is provided, but the system incurs runtime costs and delays
Solution Approach 1:
The patent performs formal verification and constraint checking during the model training and validation phases, rather than dynamically at runtime. By establishing verification rules and constraints beforehand, the system ensures safety assurance without incurring continuous runtime overhead.
Solution Approach 2:
The patent extracts and separates the verification logic from the runtime execution path. The formal verification module operates independently during model development and validation, allowing the runtime system to execute without continuous checking delays while maintaining safety guarantees.
3Reliability
If a bounding envelope or wrapper is used to check for violations dynamically, then safety checks are performed, but the system results in larger-than-necessary safety margins
Solution Approach 1:
The patent uses formal verification to precisely determine the actual safety margins required by the machine learning model based on its specific behavior and constraints. By analyzing the model's true operational characteristics rather than applying generic conservative bounds, the system achieves tight, necessary safety margins without excessive overhead.
Data Source
AI summary
Methods and systems of using machine learning to create a trusted model that improves the operation of a computer system controller are provided herein. In some embodiments, a machine learning method includes training a model using input data, extracting the model, and determining whether the model satisfies the trust-related constraints. If the model does not satisfy the trust-related constraint, modifying at least one of: the model using one or more model repair algorithms, the input data using one or more data repair algorithms, or a reward function of the model using one or more reward repair algorithms, and re-training the model using at least one of the modified model, the modified input data, or the modified reward function. If the model satisfies the trust-related constraints, providing the model as a trusted model that enables a computer system controller to perform system actions within predetermined guarantees.