Neural Network Sparsification for Out-of-Distribution Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning systems face challenges in accurately classifying out-of-distribution (OOD) data samples, as they are often trained only for pre-defined classes, leading to inaccurate or erroneous outputs when encountering inputs beyond these classes, due to the vast randomness and scale of unseen data.
Innovation Solution
The proposed solution involves preprocessing input samples by adding Gaussian noise to satisfy the Lipschitz property, using sparsification techniques to reduce feature dimensions, and generating OOD training samples by modifying in-domain samples, allowing the neural network to better distinguish between in-distribution and OOD inputs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the neural network is trained only for pre-defined classes, then the classification output for in-distribution inputs is accurate, but the classification output for out-of-distribution inputs becomes highly inaccurate and erroneous
Solution Approach 1:
The system performs preliminary action by generating synthetic OOD training samples from in-domain data before actual OOD inputs are encountered. This preparatory step creates a training dataset that includes both in-domain and synthesized OOD samples, enabling the network to learn discrimination patterns in advance rather than failing when OOD inputs first appear.
Solution Approach 2:
The system changes parameters by modifying in-domain training samples through additive noise injection and feature space transformations to create OOD samples. This parameter transformation approach generates diverse OOD variations that teach the network to recognize when inputs deviate from the true data distribution, improving both ID and OOD classification accuracy.
2Measurement precision
If the neural network is trained to identify OOD inputs, then OOD detection accuracy improves, but the complexity of training and data processing increases due to the vast randomness and scale of unseen datasets
Solution Approach 1:
The system extracts only the essential characteristics needed for OOD detection by generating OOD samples through controlled transformations of in-domain data rather than using vast amounts of random unseen data. This extraction approach captures the fundamental differences between ID and OOD inputs without requiring complex processing of large-scale diverse datasets.
Solution Approach 2:
The system creates copies of in-domain training samples and modifies them to serve as OOD training data. By copying and transforming existing labeled data rather than requiring new unlabeled OOD data, the system reduces training complexity while maintaining effective OOD detection capability.
3Reliability
If preprocessing is applied to satisfy Lipschitz property, then robustness to small input changes improves, but computational overhead increases
Solution Approach 1:
The Lipschitz constraint is applied during the preliminary training phase rather than being enforced repeatedly during inference. By incorporating the constraint into the loss function during training, the network learns robust features in advance, reducing the need for heavy computational overhead during actual classification operations.
Data Source
AI summary
An embodiment proposed herein uses sparsification techniques to train the neural network with a high feature dimension that may yield desirable in-domain detection accuracy but may prune away dimensions in the output that are less important. Specifically, a sparsification vector is generated based on Gaussian distribution (or other probabilistic distribution) and is used to multiply with the higher dimension output to reduce the number of feature dimensions. The pruned output may be then used for the neural network to learn the sparsification vector. In this way, out-of-distribution detection accuracy can be improved.