Maximum Deviation Measurement for AI Model Sufficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Assessing the sufficiency of artificial intelligence (AI) models in high-stakes applications like mortgage lending and employment practices is challenging due to the large number of possible inputs, making it difficult to determine the accuracy and reliability of AI systems.
Innovation Solution
A system that measures the maximum deviation of a supervised learning model from a reference model over a certification set and determines sufficiency based on this deviation, identifying inputs that lead to substantial deviations and preventing unexpected outputs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If comprehensive testing of AI models is performed across all possible inputs, then measurement precision is improved, but device complexity and loss of time increase significantly
Solution Approach 1:
The patent extracts a representative subset of inputs (certification set) from the complete input space, focusing testing efforts on the most critical and diverse cases rather than exhaustively testing all possible inputs. This extraction approach maintains measurement precision while reducing computational complexity.
Solution Approach 2:
The patent performs preliminary analysis to identify and select the certification set of representative inputs before conducting the actual deviation measurement. This preliminary action of selecting critical test cases in advance reduces the overall complexity and time required for comprehensive model assessment.
2Measurement precision
If comprehensive testing of AI models is performed across all possible inputs, then measurement precision is improved, but loss of time increases significantly
Solution Approach 1:
The patent extracts a representative subset of inputs (certification set) from the complete input space, focusing testing efforts on the most critical and diverse cases rather than exhaustively testing all possible inputs. This extraction approach maintains measurement precision while reducing computational complexity.
Solution Approach 2:
The patent performs preliminary analysis to identify and select the certification set of representative inputs before conducting the actual deviation measurement. This preliminary action of selecting critical test cases in advance reduces the overall complexity and time required for comprehensive model assessment.
3Reliability
If maximum deviation measurement is performed over a certification set, then reliability is improved, but device complexity increases
Solution Approach 1:
The patent introduces a reference model as an intermediary to facilitate the reliability assessment of the supervised learning model. By comparing the supervised model's outputs against the reference model's outputs on the certification set, the system can measure maximum deviation and assess reliability without requiring complex direct validation of absolute correctness.
Data Source
AI summary
Techniques regarding determining sufficiency of one or more machine learning models are provided. For example, one or more embodiments described herein can comprise a system, which can comprise a memory that can store computer executable components. The system can also comprise a processor, operably coupled to the memory, and that can execute the computer executable components stored in memory. The computer executable components can comprise a measurement component that measures maximum deviation of a supervised learning model from a reference model over a certification set and an analysis component that determines sufficiency of the supervised learning model based at least in part on the maximum deviation.