Deep Belief Network Training via Layered Expansion and Weight Transfer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep Belief Networks (DBNs) face challenges such as slow learning, getting stuck in local minima, and requiring extensive resources and data for training, especially when dealing with complex and multi-layered input data, which can lead to bottlenecks and inefficiencies in training deep layered networks.
Innovation Solution
A method is introduced to train a DBN by initially training a simpler network, expanding its width and depth, and then transferring weights to a more complex network structure, utilizing contrastive divergence gradient approximation and reconstruction error calculations to optimize training, thereby reducing resource and time requirements while maintaining or improving accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional neural networks are used for deep layered networks, then learning speed is slow and training becomes stuck in local minima, but increasing network depth and complexity to improve performance leads to slower learning and higher computational costs
Solution Approach 1:
The patent applies preliminary action by pre-training shallow belief networks layer-by-layer before combining them into a deep network. Each layer is trained independently on the training data before being stacked with other layers, allowing the network to learn features progressively without the slow convergence issues of traditional deep networks trained end-to-end
Solution Approach 2:
The deep belief network is segmented into multiple shallow belief network layers that are trained separately and then stacked. This segmentation allows each layer to learn specific features independently, improving overall learning speed while maintaining the ability to handle complex deep layered data through the hierarchical structure
2Reliability
If deep layered networks are trained to improve accuracy on complex data, then more training data and computational resources are required, but this increases hardware costs and training time
Solution Approach 1:
The network structure and weights are determined through preliminary unsupervised pre-training of shallow belief networks before final supervised training. This preliminary action allows the network to learn useful feature representations without requiring extensive labeled training data or complex hardware, reducing both data and computational resource requirements while maintaining high classification accuracy
3Reliability
If more neurons are added to increase network capacity, then better performance on complex data is achieved, but training time and computational resources increase significantly
Solution Approach 1:
The patent determines optimal network architecture and pre-trains the network structure before final training, allowing the network to learn efficient feature representations with fewer neurons. This preliminary structuring reduces the need for extensive training time while achieving better performance, as the network is already optimized for the specific task before full training begins
Solution Approach 2:
The patent changes the training parameters by using unsupervised pre-training followed by supervised fine-tuning, rather than traditional supervised training from scratch. This parameter change in the training approach allows the network to achieve better performance with reduced training time, as the pre-training phase quickly establishes useful feature hierarchies without requiring extensive computational resources
Data Source
AI summary
A computer-implemented method comprising: training a first deep belief network (DBN) using training data; adding at least one neuron to the first DBN to generate a second DBN; assigning weights to neurons in the second DBN according to weights of neurons in the trained first DBN; and training the second DBN using the training data.