Classifier Error Rate Configuration via Threshold Lookup Tables
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning classifiers are typically evaluated solely on accuracy, neglecting other important factors such as false positive and false negative rates, which users may need to customize for specific applications.
Innovation Solution
A method and system that train a first classifier to predict when a second classifier will issue classification errors, generating data buckets and threshold lookup tables based on validation sets, allowing users to select a target error rate for false positives or false negatives, and adjust thresholds accordingly to achieve desired error rates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a classifier is trained to maximize accuracy, then the overall correct classification rate is improved, but the ability to control false positive and false negative rates is lost
Solution Approach 1:
The system performs preliminary actions by training a first classifier to predict error rates before actual classification occurs. Threshold lookup tables are pre-computed for different data buckets, enabling subsequent classification to achieve target error rates without retraining. This preliminary preparation resolves the contradiction by embedding error rate control capabilities during the training phase that can be flexibly applied during deployment.
Solution Approach 2:
A first classifier is introduced as an intermediary between the input data and the second classifier. This first classifier predicts when the second classifier will issue classification errors, enabling control over false positive and false negative rates. The intermediary classifier acts as a gatekeeper that can adjust threshold selection based on desired error rates, thus resolving the contradiction between accuracy maximization and error rate customization.
2Ease of operation
If a single threshold is used for classification, then the classifier operation is simple, but the ability to achieve different target error rates is limited
Solution Approach 1:
The system segments the classification space into multiple data buckets based on input characteristics. Each data bucket has its own threshold lookup table with pre-computed thresholds for different target error rates. This segmentation allows the system to maintain simple threshold-based operation within each bucket while providing adaptability across different error rate requirements by selecting appropriate buckets and thresholds.
Solution Approach 2:
The system implements dynamic threshold selection based on the desired target error rate. Instead of using a fixed threshold, the threshold is dynamically chosen from lookup tables based on the specific error rate requirements. This dynamic adaptation resolves the contradiction by maintaining simple threshold-based operation while enabling flexible error rate control through selective threshold application.
Data Source
AI summary
A computer-implemented method comprises training, using a validation set of input, a first classifier to predict when a second classifier will issue a classification error on a particular input, the first classifier generating a number of data buckets based on the validation set of input and populating a threshold lookup table for each data bucket based on a number of thresholds set for the second classifier during the training; storing each threshold lookup table in memory; obtaining a target error rate; obtaining a new input and running the new input through the first classifier, the first classifier selecting one of the data buckets for the new input; and selecting a threshold for the second classifier using the stored threshold lookup table for the selected data bucket and the target error rate.