Tensor Deep Stacked Neural Network Parallel Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks face challenges in scalability and parallelization due to the difficulty in distributing large models across multiple machines for minibatch training, limiting their learning algorithms and efficiency.
Innovation Solution
The Tensor Deep Stacked Neural Network (T-DSN) architecture enhances the Deep Stacked Neural Network (DSN) by using a bilinear model with a tensor representation to capture higher-order statistics, shifting computational costs to convex optimization, and allowing parallel implementation on CPU clusters through a Khatri-Rao product-based hidden layer combination.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If deep neural networks use traditional training algorithms, then prediction accuracy can be achieved, but scalability and parallelization are limited due to difficulty in distributing large models across multiple machines
Solution Approach 1:
The network is divided into multiple independent blocks, each containing a hidden layer and output layer. Each block can be trained independently using batch-mode algorithms, enabling parallelization across multiple machines while maintaining overall network functionality and prediction accuracy.
Solution Approach 2:
The patent introduces a specific architecture with distinct hidden layers and output layers that act as intermediaries, allowing the network to be trained in a staged manner. This intermediary structure enables parallel training of different blocks while preserving the ability to achieve high prediction accuracy through coordinated learning.
2Adaptability or versatility
If deep stacked neural networks use non-linear hidden layers and non-linear output layers, then modeling capability is enhanced, but computational complexity increases and parallelization becomes more difficult
Solution Approach 1:
The patent applies different layer types in specific locations: non-linear hidden layers for feature transformation and a linear output layer for prediction. This localized application of non-linearity maintains modeling capability where needed while reducing computational complexity in the output stage, facilitating parallelization.
Solution Approach 2:
The patent changes the parameter structure by using a linear output layer instead of a non-linear one, and by organizing weights into block-specific parameters that can be independently optimized. This parameter reorganization reduces computational complexity while preserving the network's ability to model complex relationships through the non-linear hidden layers.
3Productivity
If deep stacked neural networks use batch-mode algorithms, then parallelization is improved, but handling of large models across multiple machines becomes more challenging
Solution Approach 1:
The network is segmented into independent blocks that can each be trained using batch-mode algorithms on separate machines. This segmentation allows parallelization across multiple processors while keeping each training task manageable in size, avoiding the complexity of distributing entire large models across the network.
Data Source
AI summary
A tensor deep stacked neural (T-DSN) network for obtaining predictions for discriminative modeling problems. The T-DSN network and method use bilinear modeling with a tensor representation to map a hidden layer to the predication layer. The T-DSN network is constructed by stacking blocks of a single hidden layer tensor neural network (SHLTNN) on top of each other. The single hidden layer for each block then is separated or divided into a plurality of two or more sections. In some embodiments, the hidden layer is separated into a first hidden layer section and a second hidden layer section. These multiple sections of the hidden layer are combined using a product operator to obtain an implicit hidden layer having a single section. In some embodiments the product operator is a Khatri-Rao product. A prediction is made using the implicit hidden layer and weights, and the output prediction layer is consequently obtained.