Deep Neural Network Training Using Extreme Learning Layers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for training and learning in deep Convolutional Neural Networks (CNNs) are computationally intensive and require large amounts of data and resources, especially when dealing with complex image processing tasks, which can be challenging for devices with limited computational power.
Innovation Solution
The implementation of improved training and learning techniques, such as extreme learning methods and Gaussian process regression, which reduce the need for extensive parameter updates and data, allowing for efficient training and enhanced generalization capabilities, particularly by designating certain layers as extreme learning layers with fixed parameters and using similarity measurements between inputs and labels.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional training methods for deep CNNs are used, then the network can achieve good performance on image processing tasks, but the computational intensity and resource requirements become excessively large
Solution Approach 1:
The patent segments the neural network into two distinct types of layers: extreme learning layers with fixed parameters that perform feature extraction, and standard training layers with learnable parameters that perform classification. This segmentation allows the computationally intensive feature extraction to be done once with fixed parameters, while subsequent classification layers can be trained efficiently with standard methods, significantly reducing overall computational resource requirements.
Solution Approach 2:
The patent applies preliminary action by pre-training the extreme learning layers with fixed parameters before the standard training layers. The extreme learning layers are trained once to extract meaningful features from input data, and these pre-extracted features are then used by the standard training layers for classification. This preliminary feature extraction eliminates the need for the entire network to undergo extensive training, reducing computational intensity while maintaining performance.
2Adaptability or versatility
If the number of layers and nodes in the CNN is increased to handle complex tasks, then the network's capability improves, but the training time and data requirements increase significantly
Solution Approach 1:
The patent divides the network into extreme learning layers with fixed parameters and standard training layers with learnable parameters. The extreme learning layers perform universal feature extraction that can be reused across different tasks, while the standard training layers are trained specifically for each classification problem. This segmentation allows deep networks to maintain high capability while reducing training time, as only the standard training layers require extensive training for each specific task.
Solution Approach 2:
The patent changes the parameter status of certain layers from learnable to fixed. By designating extreme learning layers with fixed parameters, the network can maintain deep architecture for high capability while avoiding the computational burden of training all parameters. The fixed parameters in extreme learning layers are determined once during pre-training and remain constant during subsequent classification tasks, significantly reducing training time and data requirements.
3Reliability
If more training data is used to improve model accuracy, then the network performance improves, but the data processing burden and computational cost increase
Solution Approach 1:
The patent applies preliminary action by pre-training extreme learning layers to extract robust features from training data once. These pre-extracted features capture the essential patterns and are stored as fixed parameters. When new data comes in for classification tasks, the system only needs to process this new data through the standard training layers using the pre-extracted features, rather than re-processing all training data. This dramatically reduces the data processing burden while maintaining high model accuracy.
Solution Approach 2:
The patent creates a copy of the training process in the extreme learning layers that is performed once with fixed parameters. These layers learn to extract features from training data and store this knowledge as fixed parameters. Subsequent classification tasks then use these pre-learned features as a template or copy, allowing the network to achieve high accuracy on new data without re-processing the original training data, thus reducing the quantity of data that needs to be actively processed.
4Productivity
If conventional training methods are used on devices with limited computational power, then the network can process data, but the training efficiency and speed are severely limited
Solution Approach 1:
The patent segments the network into extreme learning layers with fixed parameters that perform feature extraction and standard training layers with learnable parameters that perform classification. This segmentation enables the system to leverage the limited computational power of edge devices for the classification task while the computationally intensive feature extraction is handled by more powerful servers or clouds. The fixed parameters in extreme learning layers can be efficiently loaded and executed on resource-constrained devices, significantly improving training efficiency and speed compared to conventional methods that require extensive training on every device.
Data Source
AI summary
Methods and systems are disclosed using improved training and learning for deep neural networks. In one example, a deep neural network includes a plurality of layers, and each layer has a plurality of nodes. The nodes of each L layer in the plurality of layers are randomly connected to nodes of an L+1 layer. The nodes of each L+1 layer are connected to nodes in a subsequent L layer in a one-to-one manner. Parameters related to the nodes of each L layer are fixed. Parameters related to the nodes of each L+1 layers are updated. In another example, inputs for the input layer and labels for the output layer of a deep neural network are determined related to a first sample. A similarity between different pairs of inputs and labels is estimated using a Gaussian regression process.