Classification Model Training with Neighbor Consistency Regularization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current label propagation methods for deep learning are prone to underfitting and are computationally expensive, and they do not effectively utilize noisy labels available from sources like social media and webpages.
Innovation Solution
A computer-implemented method for training a classification model that involves obtaining a training dataset, processing inputs with an encoder model to generate embeddings, and using a classification model to generate classifications. The method determines a similarity measure between embeddings and evaluates a loss function that includes a term penalizing the divergence of predicted classifications from weighted combinations of neighbor classifications, to adjust the model parameters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If current label propagation methods are used, then labels can be propagated based on learned feature representation, but underfitting occurs and computational cost increases
Solution Approach 1:
The patent segments the global graph into multiple local graphs, where each local graph contains a sample and its k-nearest neighbors. This segmentation reduces the computational complexity from O(n²) in global graphs to O(kn) in local graphs, while still capturing important local structures for accurate label propagation.
Solution Approach 2:
The patent introduces a regularization term in the loss function that operates in the embedding space dimension, constraining the embeddings of neighboring samples to be similar. This adds a new dimension of control that improves label propagation reliability without requiring full global graph computation.
2Reliability
If current label propagation methods are used, then labels can be propagated across the dataset, but overfitting and underfitting of labels cause misclassification issues
Solution Approach 1:
The patent modifies the loss function by adding a regularization term that changes the parameter constraints on embedding similarity. This parameter change enforces label consistency by ensuring that neighboring samples in the embedding space have similar predictions, reducing both overfitting and underfitting.
Solution Approach 2:
The patent implements a feedback mechanism where the loss function continuously evaluates the similarity between neighboring embeddings and adjusts the model parameters to maintain consistency. This feedback loop ensures that label propagation remains stable and accurate across iterations.
3Loss of information
If global graphs are constructed for label propagation, then comprehensive label information can be propagated, but computational expense increases significantly
Solution Approach 1:
The patent extracts only the necessary local neighborhood information from the global graph structure, taking out the k-nearest neighbors for each sample. This extraction approach captures the essential local label propagation relationships without the computational burden of constructing and processing the entire global graph.
Data Source
AI summary
Systems and methods for classification model training can use feature representation neighbors for mitigating label training overfitting. The systems and methods disclosed herein can utilize neighbor consistency regularization for training a classification model with and without noisy labels. The systems and methods can include a combined loss function with both a supervised learning loss and a neighbor consistency regularization loss.