Training Data Outlier Scoring for Inconsistent Annotation Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional outlier detection methods fail to reliably identify inconsistent data annotations in machine learning datasets, leading to erroneous outputs and cumbersome mitigation actions.
Innovation Solution
A system that analyzes data points in a joint input-output space to determine input and output outlier scores, generating a total outlier score to identify and exclude inconsistent data points, thereby improving the quality of training datasets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional outlier detection methods are used, then the detection process is simple, but inconsistent data annotations cannot be reliably identified
Solution Approach 1:
The patent combines input outlier detection and output outlier detection into a unified framework that evaluates data points in both input and output spaces simultaneously. This merging allows the system to reliably identify inconsistent annotations by considering both input feature deviations and output label deviations together, rather than relying on either method alone.
Solution Approach 2:
The patent extends outlier detection from a single-dimensional input space to a joint input-output space by introducing output outlier scores. This dimensional expansion enables the detection system to capture inconsistencies that manifest in the output labels while maintaining their relationship to input features, thereby improving detection reliability without excessive complexity.
2Measurement precision
If conventional outlier detection methods are used, then the processing speed is fast, but detection accuracy is insufficient for vast and intricate datasets
Solution Approach 1:
The patent segments the outlier detection process into distinct components: input outlier score calculation, output outlier score calculation, and combined assessment. This segmentation allows each component to be optimized independently while maintaining overall efficiency, enabling accurate detection in vast datasets without sacrificing processing speed.
Solution Approach 2:
The patent performs preliminary outlier score calculations for both input and output dimensions before making final detection decisions. This preliminary action allows the system to pre-compute necessary metrics, reducing the computational burden during final detection and maintaining productivity while achieving higher accuracy.
3Reliability
If inconsistent data annotations are not identified, then the training process is efficient, but model performance is compromised
Solution Approach 1:
The patent implements a self-service mechanism where the detection system automatically identifies and flags inconsistent data annotations without requiring manual review. This automation maintains model performance reliability by systematically detecting issues while minimizing the time loss associated with manual data preprocessing and verification.
4Measurement precision
If comprehensive outlier detection is performed, then detection thoroughness is high, but computational resources are consumed
Solution Approach 1:
The patent applies partial action by focusing computational resources on calculating outlier scores only for data points that show deviations in either input or output dimensions. Rather than performing exhaustive analysis on all data points uniformly, the system selectively intensifies detection efforts where anomalies are likely, achieving thorough detection with reduced energy consumption.
Data Source
AI summary
Systems and methods are disclosed for generating training data for a machine learning model. The method includes receiving, from one or more data sources, a first machine learning training data set that includes a plurality of data points; determining an input outlier score of a first data point of the plurality of data points; determining an output outlier score of the first data point; generating a total output score of the first data point based on the input outlier score of the first data point and the output outlier score of the first data point, the total output score of the first data point representing a likelihood that the first data point is an inconsistently annotated data point; comparing the total output score of the first data point with a pre-determined threshold; based on the comparison, generating a second machine learning training data set that excludes the first data point.