AI Label Sampling with Heuristic k-NN Noise Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models face challenges with inaccurate or unreliable data labels, particularly in scenarios where datasets are largely unlabeled with a small quantity of noisy labels, leading to inefficiencies and inaccuracies in performance.
Innovation Solution
A computing platform applies undersampling and a modified k-Nearest Neighbors (k-NN) algorithm with heuristic scoring to filter out unreliable data points, using business rules to rank data and identify trustworthy labels, thereby improving the accuracy and reliability of machine learning models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional data preparation methodologies (OCC, SMOTE) are used, then data processing can be performed, but the model accuracy and reliability deteriorate due to noisy and imbalanced labels
Solution Approach 1:
The system performs preliminary actions by computing heuristic scores for all data points before the actual labeling process. This pre-computation of trustworthiness scores allows the system to identify and filter noisy labels in advance, ensuring that only high-quality labels are used for model training, thereby resolving the contradiction between processing speed and label quality
Solution Approach 2:
The system enables self-service by allowing data points to evaluate their own trustworthiness through heuristic scoring. Each data point is automatically assessed based on its characteristics and compared with neighboring points, eliminating the need for manual label verification and enabling the system to self-correct labeling errors, thus improving both reliability and precision
2Manufacturing precision
If manual labeling by subject matter experts is performed, then label accuracy improves, but processing time and cost increase significantly
Solution Approach 1:
The system introduces an intermediary mechanism - the heuristic scoring system with k-NN comparison - that acts as a bridge between automated labeling and manual expert labeling. Instead of relying solely on expensive manual labeling or crude automated methods, the intermediary score computation evaluates label trustworthiness objectively, achieving high accuracy without the time and cost penalties of manual expert review
Solution Approach 2:
The system replaces the mechanical process of manual expert labeling with an automated computational system. The heuristic scoring mechanism substitutes human experts' judgment with algorithmic evaluation that compares data points against their k nearest neighbors, achieving comparable or superior accuracy while eliminating the time loss associated with manual labeling processes
3Productivity
If generic quantitative similarity methods are used for classification, then processing speed improves, but classification accuracy deteriorates due to inability to handle noisy labels
Solution Approach 1:
The system changes the parameter used for classification from generic quantitative similarity to a trustworthiness score based on heuristic evaluation and k-NN comparison. This parameter transformation allows the system to maintain high processing speed while achieving superior classification accuracy by filtering out noisy labels that would otherwise corrupt the classification results
Solution Approach 2:
The system implements feedback by using the heuristic scores and k-NN comparisons to continuously evaluate and filter labels during the processing pipeline. This feedback mechanism ensures that only high-confidence labels are used for classification, maintaining both processing speed and accuracy by dynamically adjusting which data points influence the final classification
Data Source
AI summary
Aspects of the disclosure relate to smart sampling of noisy labels using artificial intelligence. A computing platform may receive a dataset of primarily unlabeled data points. The computing platform may apply undersampling to the unlabeled data points to reduce imbalance. The computing platform may assign a candidate label to each unlabeled data point in the dataset without a human manually labeling the unlabeled data points. The computing platform may compute a heuristic score for each data point and rank the data points based on the heuristic score. The computing platform may subsample the dataset by comparing the heuristic score for each data point against more than one threshold and applying a k-Nearest Neighbors (k-NN) algorithm, with two different k values, to identify untrustworthy labels. The computing platform may provide or transmit a trustworthy resulting dataset to a machine learning (ML) model.