Neural Network Training Data Reduction via Loss Comparison
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for training neural networks are inefficient due to high energy consumption and data movement costs, particularly during decentralized training, where not all data transferred has an impact on the target variable, and significant energy is expended on unnecessary data processing.
Innovation Solution
A method that involves performing forward and backward passes on input data sets to calculate losses, followed by quantized passes to determine if the data set can be reduced, thereby minimizing data transfer and processing by identifying and removing non-essential features, using techniques like PCA or autoencoders to create a reduced data set.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If batches of data are moved from non-volatile memory to RAM/GPU memory for training, then the neural network can be trained, but significant energy is consumed and time is wasted transferring data that may not impact the target variable
Solution Approach 1:
The patent extracts and removes non-essential features from the input data set, retaining only the features that have an impact on the target variable. This is achieved by comparing the loss values from regular and quantized forward passes, and when the difference is below a threshold, the data set is reduced by removing redundant features, thereby reducing energy consumption during data transfer and processing
Solution Approach 2:
The patent changes the parameters of the data set by quantizing the input data and comparing the loss values between regular and quantized passes. When the loss difference is within a threshold, the data set parameters are modified to reduce the number of features, optimizing the balance between energy consumption and training effectiveness
2Reliability
If all features of the input data are processed, then complete information is available for training, but energy is wasted on features that do not impact the target variable
Solution Approach 1:
The patent extracts and removes non-essential features from the input data set, retaining only the features that have an impact on the target variable. This is achieved by comparing the loss values from regular and quantized forward passes, and when the difference is below a threshold, the data set is reduced by removing redundant features, thereby reducing energy consumption during data transfer and processing
Solution Approach 2:
The patent uses feedback from loss value comparisons between regular and quantized passes to determine whether to reduce the data set. The feedback mechanism allows the system to adaptively decide which features to retain based on their actual impact on the target variable, optimizing energy consumption while maintaining training accuracy
3Adaptability or versatility
If data is transferred over a computer network for decentralized training, then distributed computation is enabled, but significant time and energy are consumed for data movement
Solution Approach 1:
The patent extracts and removes non-essential features from the input data set before transfer, reducing the amount of data that needs to be moved across the network. This decreases both the time and energy required for data transfer while maintaining the decentralized training capability through the reduced data set
Solution Approach 2:
The patent applies partial action by transferring only the essential features of the data set rather than all features. This selective transfer approach reduces the data movement overhead in decentralized training while preserving the necessary information for effective model training
Data Source
AI summary
Embodiments described herein relate to methods and apparatuses for training a neural network. A method comprises receiving an input data set at a layer of the neural network; performing a forward pass and a backward pass on the input data set to determine regular output data; calculating a first loss associated with the regular output data; performing a quantized forward pass and a quantized backward pass on the input data set to determine quantized output data; calculating a second loss associated with the quantized output data; comparing the first loss to the second loss; and based on the comparison determining whether to reduce the input data set to provide a reduced data set.