Machine Learning Function Testing Through Reference-Metric Interpolation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing functions, particularly machine learning models, often produce indeterminate performance due to false positives and false negatives, leading to unsuitable or ineffective outcomes in tasks such as classification and regression, resulting in a 'dirty population' of incorrectly labeled records.
Innovation Solution
A testing engine employs multiple machine learning models to assess function performance by comparing outputs against reference functions with known metrics, using a first model trained on core features and a second model on additional predictive features to determine mislabeled records, enabling interpolation of performance metrics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a single machine learning model is used to evaluate function performance, then the evaluation process is simple and fast, but the measurement precision and reliability of performance assessment are insufficient
Solution Approach 1:
The evaluation process is segmented into multiple independent machine learning models, each trained to detect mislabeled records from different perspectives. The first model evaluates based on primary features while the second model evaluates based on additional features, allowing comprehensive assessment without requiring a single complex model
Solution Approach 2:
Multiple machine learning models serve as intermediaries between the function output and the final performance metric. Each model acts as an independent evaluator that processes the function output and provides a mislabeled record count, which are then aggregated to determine the overall performance metric
2Reliability
If multiple machine learning models are applied to assess mislabeled records, then the performance evaluation becomes more accurate, but the computational time and processing complexity increase
Solution Approach 1:
The machine learning models are pre-trained on training datasets before deployment. This preliminary training action ensures that when the models are applied to evaluate function outputs, they can quickly and accurately identify mislabeled records without requiring extensive computation during the actual evaluation phase
Solution Approach 2:
The system applies multiple models (excessive action) to ensure comprehensive and reliable evaluation. Rather than using a single model that might miss certain types of errors, multiple models work together to cover different aspects of performance assessment, with their results aggregated to produce a robust performance metric
Data Source
AI summary
A method for determining the performance metric of a function may include interpolating the performance metric of the function relative to a known performance metric of a reference function. The performance metric of the function may be interpolated based on a first difference in a performance of the function measured by applying a first machine learning model and a performance of the function measured by applying a second machine learning model. The performance metric of the function may be further interpolated based on a second difference in a performance of the reference function measured by applying the first machine learning model and a performance of the reference function measured by applying the second machine learning model. The function may be deployed to a production system if the performance metric of the function exceeds a threshold value. Related systems and articles of manufacture, including computer program products, are also provided.