Chaining ML Models Using Output Confidence Levels
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing AI/ML frameworks face challenges in efficiently training and deploying machine learning models due to resource-intensive computations and the need for large amounts of training data, which is costly and difficult to obtain.
Innovation Solution
The proposed solution involves chaining machine learning models using a confidence level associated with the output of a publisher model, allowing the consumer model to process outputs based on the confidence level, thereby improving accuracy and reducing the need for joint retraining of models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If machine learning models are chained without using confidence levels, then the system is simpler and faster, but the accuracy and reliability of the output decrease
Solution Approach 1:
The publisher model generates confidence levels as part of its output before the consumer model processes the data. This preliminary action of calculating confidence levels allows the consumer model to make informed decisions about which inputs to process further, improving overall system reliability without requiring complex post-processing
Solution Approach 2:
Instead of uniformly processing all outputs from the publisher model, the consumer model selectively processes only those outputs that meet certain confidence level thresholds. This local quality approach applies different processing strategies to different portions of the data based on their individual confidence levels, improving accuracy while maintaining efficiency
2Reliability
If publisher and consumer models are trained together as a joint system, then the models are optimized for each other, but the training cost and computational resources increase significantly
Solution Approach 1:
The training process is segmented into independent phases: the publisher model is trained first to generate outputs with confidence levels, and then the consumer model is trained separately using those outputs as input. This segmentation allows each model to be optimized independently, reducing the computational burden of joint training while maintaining model compatibility through the confidence level interface
Solution Approach 2:
The publisher model is trained in advance to produce well-calibrated confidence levels before the consumer model training begins. This preliminary action ensures that the consumer model receives high-quality, pre-processed input with reliable confidence information, improving its training efficiency and performance without requiring simultaneous joint training
3Measurement precision
If large amounts of training data are used to achieve high accuracy, then the model performance improves, but the cost and difficulty of obtaining and processing the data increase
Solution Approach 1:
The system changes the parameter of confidence level calibration in the publisher model to improve the quality of outputs. By optimizing the confidence level distribution and calibration, the consumer model can achieve high accuracy with fewer training examples, as the confidence levels provide additional supervisory signal that reduces the need for large volumes of labeled training data
Data Source
AI summary
Systems and techniques are provided for chaining machine learning (ML) models using a confidence level of an output of a provider model. An example method can include receiving a first output generated by a first ML model, determining a confidence level of the first output generated by the first ML model, and providing the first output of the first ML model and the confidence level of the first output to a second ML model as an input of the second ML model. The second ML model can be configured to process the first output of the first ML model and the confidence level of the first output of the first ML model to generate a second output of the second ML model.