Outlier Identification in Neural Network Training Datasets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The proliferation of data and human intervention in labeling machine learning datasets lead to incorrectly labeled items, affecting the performance of machine learning systems, which is costly and time-consuming to rectify through manual review.
Innovation Solution
A method and system for automatically identifying outliers in a neural network training dataset by generating embedding vectors, calculating dissimilarity scores, and removing elements with scores above a predetermined threshold, while using a generative convolutional neural network to augment the dataset if necessary.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual review is used to identify incorrectly labeled items, then data quality is improved, but cost and time consumption increase significantly
Solution Approach 1:
The patent replaces the mechanical manual review process with an automated computational system. The system uses machine learning models to generate embedding vectors for training data items, calculates dissimilarity scores by comparing each item's embedding to the centroid of the dataset, and automatically identifies outliers. This substitution eliminates the need for human reviewers while maintaining the ability to detect incorrectly labeled items, thereby resolving the contradiction between data quality improvement and time consumption reduction.
Solution Approach 2:
The system enables the training dataset to self-identify its own outliers through automated computational processes. By using embedding vectors and dissimilarity score calculations, the dataset can autonomously flag potentially incorrect items without external human intervention. This self-service mechanism allows continuous automated monitoring and identification of data quality issues, resolving the contradiction by eliminating manual review time while preserving reliability.
2Reliability
If human intervention is used for labeling, then data accuracy is improved, but cost increases due to human labor
Solution Approach 1:
The patent substitutes expensive human labeling and review processes with automated machine learning-based embedding and dissimilarity calculation systems. The automated system processes training data items through neural network embeddings and mathematical distance calculations to identify outliers, eliminating the need for continuous human labor while maintaining data accuracy through algorithmic detection of incorrectly labeled items.
Solution Approach 2:
The system creates computational representations (embedding vectors) of training data items that capture their essential characteristics. These vector copies allow the system to analyze and compare items efficiently without requiring human reviewers to examine each original item, thereby reducing labor costs while preserving the ability to detect labeling errors through the vector-based analysis.
3Productivity
If automated tools are used for data collection, then productivity is improved, but data quality deteriorates due to human error and malicious disturbance
Solution Approach 1:
The patent applies preliminary action by automatically identifying and flagging potential outliers in the training dataset before the data is used for model training. The system calculates embedding vectors and dissimilarity scores in advance to detect incorrectly labeled items, allowing data quality issues to be addressed proactively rather than discovering them during or after training. This preliminary detection maintains the high productivity of automated data collection while improving reliability by filtering out problematic data points.
Solution Approach 2:
The system implements feedback by using the dissimilarity score calculations to identify outliers that may represent human errors or malicious disturbances in the training data. This feedback mechanism allows the automated data collection system to self-correct by flagging suspicious items for review or removal, thereby maintaining both the productivity of automated collection and the reliability of the final training dataset.
Data Source
AI summary
Systems and methods for automatically identifying outliers in Machine Learning training datasets. The method includes gaining access to a training set for the neural network NN. For each element of the training dataset, an embedding vector is generated, which is a numeric representation of the corresponding element. A centroid of all the embedding vectors of all the elements of the training set is computed equal to an average of all the embedding vectors of all the elements of the training set. A dissimilarity score is generated for each element of the training set by calculating a distance between the embedding vector corresponding to the element and the centroid. The method further includes identifying the elements from the training set with embedding vectors having the dissimilarity score higher than or equal to a predetermined threshold value.