AI Predictor Compatibility Evaluation Using Backward Index
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
After updating AI models, there is a challenge in ensuring compatibility between the pre-update and post-update AI predictors, leading to potential changes in behavior that may result in incorrect answers and requiring significant effort to adapt business operations.
Innovation Solution
A compatibility evaluation device and method that calculates a generalized backward compatibility index using relational expressions to assess the compatibility between pre-update and post-update predictors, providing a score that indicates the degree of coincidence in correct and incorrect answers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If AI models are updated to improve accuracy, then the prediction accuracy is improved, but the compatibility with previous AI behavior deteriorates
Solution Approach 1:
The patent applies preliminary action by calculating a compatibility index between the updated AI model and the previous model before deploying the update. This allows operators to predict potential compatibility issues in advance and take corrective measures, such as adjusting the updated model or preparing fallback plans, thereby preventing severe behavioral changes while still allowing accuracy improvements
2Adaptability or versatility
If AI models are updated frequently to adapt to environmental changes, then the adaptability is improved, but the operational stability deteriorates
Solution Approach 1:
The patent implements feedback by using the calculated compatibility index to inform future model update decisions. When the compatibility index indicates low compatibility, the system provides feedback to operators to either adjust the updated model to improve compatibility or delay the update, thus maintaining operational stability while still allowing necessary adaptations
Data Source
AI summary
In a compatibility evaluation device, the acquisition means acquires outputs of a first predictor and a second predictor for evaluation data. The index determination means determines a generalized backward compatibility index defined by a combination of a plurality of relational expressions representing relationship between the output of the first predictor and the output of the second predictor. The calculation means calculates a score indicating compatibility of the first predictor and the second predictor, using the output of the first predictor, the output of the second predictor, and the generalized backward compatibility index.