Neural Network Activation and Loss Optimization for Noisy Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks trained on noisy datasets inherit biases and errors, particularly due to incorrect, missing, or imbalanced labels, which conventional solutions like focal loss or softmax activation struggle to address effectively, especially in unstructured data.
Innovation Solution
The implementation of an optimized neural network that combines various activation and loss functions, such as sigmoid and softmax, with normalization to handle noisy labels, allowing for continuous convergence and improved robustness in noisy environments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional softmax activation and multi-class cost calculation are used, then the solution is simple to implement, but it is not robust enough for unstructured noisy data
Solution Approach 1:
The patent combines multiple activation functions (sigmoid and softmax) with normalization techniques to create a hybrid approach that leverages the strengths of each method. This merging allows the system to handle noisy labels more effectively while maintaining computational efficiency through vectorized operations.
Solution Approach 2:
The solution uses composite loss functions that integrate multiple computational approaches (sigmoid cross-entropy, softmax cross-entropy, and normalization operations) into a unified framework. This composite structure provides robustness against various types of noise while maintaining a streamlined implementation through efficient linear algebra operations.
2Measurement precision
If focal loss with additional weights and tuning is used, then classification focus on specific labels is improved, but the solution becomes limited to specifically designed applications
Solution Approach 1:
The patent develops a universal normalization-based approach that can be applied to any multi-class classification problem regardless of the specific dataset or noise characteristics. The method uses standard activation functions combined with normalization, making it broadly applicable without requiring dataset-specific tuning or custom weight parameters.
Solution Approach 2:
The solution changes the approach from adjusting classification weights (as in focal loss) to applying normalization transformations to the activation outputs. This parameter change shifts the focus from weight-based prioritization to normalization-based robustness, enabling broader applicability across different classification scenarios.
3Reliability
If standard data preparation techniques like structuring or cleansing are applied, then data quality is improved, but it is not feasible for highly varied unstructured data
Solution Approach 1:
The patent replaces mechanical data preparation processes (structuring, cleansing, validation) with a computational approach applied during model training. Instead of physically processing and cleaning the data beforehand, the normalization-based method handles noise computationally during the learning process, making it feasible for highly varied unstructured data.
Solution Approach 2:
The normalization approach enables the neural network to self-correct for noisy labels during training without requiring external data cleaning interventions. The model automatically adapts to the noise characteristics present in the data through the normalization mechanism, eliminating the need for manual or automated data preparation pipelines.
Data Source
AI summary
A system includes a computing platform having a hardware processor and a memory storing a software code and a neural network (NN) having multiple layers including a last activation layer and a loss layer. The hardware processor executes the software code to identify different combinations of layers for testing the NN, each combination including candidate function(s) for the last activation layer and candidate function(s) for the loss layer. For each different combination, the software code configures the NN based on the combination, inputs, into the configured NN, a training dataset including multiple data objects, receives, from the configured NN, a classification of the data objects, and generates a performance assessment for the combination based on the classification. The software code determines a preferred combination of layers for the NN including selected candidate functions for the last activation layer and the loss layer, based on a comparison of the performance assessments.