Neural Network Out-of-Distribution Classification via Contrastive Loss
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models struggle to accurately classify out-of-distribution network inputs, which can lead to unreliable predictions and potential failures in safety-critical applications.
Innovation Solution
A method for training a neural network using a supervised loss and a contrastive loss, which generates accurate network outputs and intermediate outputs that can reliably classify out-of-distribution network inputs, without requiring access to labeled out-of-distribution data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a neural network is trained using only supervised loss on in-distribution data, then the network achieves good accuracy on training tasks, but the network produces unreliable predictions on out-of-distribution inputs
Solution Approach 1:
The patent segments the training objective into two distinct components: supervised loss for accurate classification and contrastive loss for distribution discrimination. By dividing the single training task into multiple specialized objectives, the network learns both accurate predictions and the ability to identify out-of-distribution inputs independently
Solution Approach 2:
The neural network is designed to perform multiple functions simultaneously: it maintains its primary classification function while also acquiring the ability to detect out-of-distribution inputs. The contrastive loss enables the network to universally handle both in-distribution and out-of-distribution cases without requiring separate models
2Reliability
If labeled out-of-distribution data is used for training, then the network can learn to classify out-of-distribution inputs, but the requirement for labeled out-of-distribution data increases training complexity and data availability constraints
Solution Approach 1:
The network uses contrastive loss to enable itself to distinguish out-of-distribution inputs without external guidance. The contrastive objective allows the network to self-learn distribution boundaries by comparing embeddings, making the training process self-sufficient without requiring labeled out-of-distribution data
Solution Approach 2:
The patent changes the training parameter from labeled out-of-distribution data to unlabeled out-of-distribution data combined with contrastive loss. This parameter change transforms the training requirement from needing annotated examples to needing only the ability to compute contrastive embeddings, significantly reducing data annotation complexity
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a neural network to (i) generate accurate network outputs for a machine learning task and (ii) generate intermediate outputs that can be used to reliably classify out-of-distribution inputs. In one aspect, a method comprises: training the neural network using supervised and contrastive losses, comprising repeatedly performing operations including: obtaining first and second network inputs; processing each network input using the neural network to generate its respective network input embedding; processing the first network input using the neural network to generate a network output; and adjusting the network parameter values using supervised and contrastive loss gradients, wherein: the supervised loss is based on: (i) the network output, and (ii) a corresponding target network output; and the contrastive loss is based on at least: (i) the first network input embedding, and (ii) the second network input embedding.