Neural Network Loss Contribution Detection Through Forward-Backward Gradients
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for detecting training data that contribute to neural network loss require enormous computational resources, leading to prolonged detection times.
Innovation Solution
A method involving the calculation of products from input data through forward and backward propagation functions to identify data contributing most to loss, followed by adjusting weights and potentially removing or retraining based on these products.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional methods are used to detect training data contributing to neural network loss, then detection accuracy is maintained, but computational requirements and time consumption increase enormously
Solution Approach 1:
The patent segments the loss detection problem by calculating contribution values for individual training data points separately through forward and backward propagation, rather than analyzing all data collectively. This allows identification of specific high-contribution data points without requiring exhaustive computation across the entire dataset.
Solution Approach 2:
The patent changes the parameter of analysis from global loss evaluation to individual data point contribution evaluation. By computing contribution values as products of forward propagation outputs and backward propagation gradients for each training example, the method transforms the detection parameter to enable efficient identification of problematic data points.
2Reliability
If conventional methods are used to detect training data contributing to neural network loss, then comprehensive analysis is performed, but training time increases significantly
Solution Approach 1:
The patent performs preliminary computation by calculating contribution values during the normal training process through forward and backward propagation. These contribution values are computed as part of the standard training loop, allowing subsequent identification of high-contribution data points without requiring additional separate analysis passes.
Solution Approach 2:
The training process itself generates the necessary information for loss contribution detection through its inherent forward and backward propagation mechanisms. The contribution values are derived from the same computational graphs used for weight updates, making the detection process self-contained within the existing training workflow without requiring external analysis tools.
Data Source
AI summary
An exemplary method comprises calculating products, each product being generated by multiplying a first result given by applying one of a plurality of input data used for training a machine learning model to a first function that represents a forward propagation of the plurality of input data through the machine learning model, and a second result given by applying the one of the plurality of input data to a second function that represents a backward propagation of a partial differential of a loss therethrough, the loss being generated by summing errors between desired values that the machine learning model should output in response to the plurality of input data and prediction values that the machine learning model outputs in response thereto; and detecting among the plurality of input data, input data that contributes the most to the loss, based on magnitudes of the products.