CNN Training via PCA Eigenvectors and Affinity Matrices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training deep convolutional neural networks for image classification is computationally complex and time-consuming, especially with large training sets and many layers, despite existing pooling techniques like max pooling and mean pooling.
Innovation Solution
The method employs linear principal component analysis (PCA) to reduce the number of feature vectors and eigenvectors, using a multi-layered deep cascade of neural networks for feed-forward training, generating rich feature vectors and affinity matrices to efficiently train convolutional neural networks without back-propagation, followed by conventional error back-propagation for the final layers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If back propagation is used to adjust weights through all levels of a neural network, then the network can be trained with classified data to achieve accurate object recognition, but the training process becomes computationally expensive and time-consuming
Solution Approach 1:
The patent segments the training process into two distinct phases: an unsupervised pre-training phase that processes only the first layer using PCA to generate feature vectors, and a supervised fine-tuning phase that uses back propagation only on subsequent layers. This segmentation avoids applying computationally expensive back propagation to all layers, significantly reducing training time while maintaining classification accuracy.
Solution Approach 2:
The patent performs preliminary unsupervised pre-training of the first layer using PCA to generate affinity matrices and feature vectors before applying supervised back propagation. This preliminary action prepares the network in advance, allowing the subsequent supervised training to focus only on finer adjustments, thereby reducing overall training time while preserving accuracy.
2Adaptability or versatility
If the number of hidden layers in a deep neural network is increased to learn more complex features, then the network's ability to recognize abstract patterns improves, but the computational complexity and training time increase significantly
Solution Approach 1:
The patent applies segmentation by treating the first layer differently from subsequent layers. The first layer undergoes unsupervised PCA-based pre-training to learn basic features, while deeper layers receive supervised fine-tuning. This segmented approach enables deep networks to learn complex hierarchical features without proportionally increasing training complexity.
Solution Approach 2:
The patent introduces affinity matrices and feature vectors as intermediary representations generated by PCA in the first layer. These intermediaries serve as simplified inputs to subsequent layers, reducing the dimensional complexity that would otherwise propagate through the entire deep network, thereby managing network complexity while preserving feature learning capability.
3Measurement precision
If the size of the training set is increased to improve recognition accuracy, then the quality of classification improves, but the computational resources and training time required increase
Solution Approach 1:
The patent extracts only the essential statistical properties from large training sets through PCA in the first layer, generating affinity matrices that capture the most important feature relationships. This extraction process condenses information from large datasets into compact representations, allowing the network to learn from large training sets without proportionally increasing computational power requirements for subsequent layers.
4Device complexity
If conventional pooling techniques like max pooling or mean pooling are used to reduce dimensionality, then the training process becomes more manageable, but the accuracy and efficiency of feature representation are compromised
Solution Approach 1:
The patent substitutes conventional mechanical pooling operations (max pooling, mean pooling) with a statistical approach based on PCA and affinity matrices. Instead of discarding information through pooling, the PCA-based method transforms feature vectors into affinity matrices that preserve statistical relationships, achieving dimensionality reduction while maintaining or improving feature representation accuracy.
Data Source
AI summary
A method of object recognition trains a convolutional neural network (CNN) with a set of training images, then classifies an image of an object using the trained CNN. A first layer of the CNN is trained by generating a set of first convolutional filters from eigenvectors produced from linear principal component analysis of patches of the training images. The training of each of multiple hidden layers CNN includes generating a set of convolutional filters from a selected subset of eigenvectors produced from linear principal component analysis of patches of an affinity matrix constructed using a set of prior convolutional filters from a prior layer of the CNN, where the affinity matrix represents correlations of feature vectors associated with the prior layer. The last layer of the CNN is trained with a regular classifier by error back-propagation using the training images and labels associated with the training images.