Feedforward Neural Network Pseudo-Recurrent Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Recurrent Neural Networks (RNNs) face challenges in training due to difficulties in learning long-term dependencies and convergence issues, and are computationally expensive for both training and testing, especially when dealing with incomplete data.
Innovation Solution
A pseudo-recurrent processing method is introduced, utilizing mixture models and multiple imputation, where hidden layer activities are governed by dynamical equations and cluster centers are used to replace costly sampling operations, allowing for efficient classification of incomplete data by mimicking RNN behavior in a feedforward network.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If Recurrent Neural Networks are used for processing sequential information and dealing with long-range statistical relationships, then the model's ability to capture temporal dependencies and perform data completion is improved, but the computational cost and training complexity increase significantly
Solution Approach 1:
The patent segments the recurrent processing into two distinct phases: an offline training phase where the feedforward network learns from complete data, and an online inference phase where pseudo-recurrent iterations are performed on incomplete data. This segmentation allows the complex RNN-like behavior to be achieved without maintaining the full computational burden of traditional RNNs during both training and testing
Solution Approach 2:
The patent performs preliminary training of a feedforward neural network offline using complete data samples. During this preliminary action, the network learns the underlying data distribution and relationships. This pre-learned knowledge is then reused during online processing to handle incomplete data through pseudo-recurrent iterations, avoiding the need for expensive online training
2Productivity
If traditional imputation methods are used to fill missing data before classification, then the preprocessing is simple and fast, but the accuracy of classification deteriorates due to loss of statistical information
Solution Approach 1:
The patent implements a feedback mechanism where the network's own output is fed back into the input for iterative refinement. During pseudo-recurrent processing, the network generates predictions, which are then fed back as new inputs along with the original incomplete data. This feedback loop allows the network to progressively refine its estimates and recover statistical information that would be lost in single-pass imputation methods
Solution Approach 2:
The patent changes the parameter representation by using the network's hidden layer activations and probability distributions as dynamic imputation parameters. Instead of using fixed statistical moments or simple interpolation values, the system utilizes the full probabilistic output of the trained network to generate imputed values that preserve the underlying data distribution and relationships
3Adaptability or versatility
If energy minimization and sampling methods are used in RNNs for data generation and completion, then the generative capability is improved, but the computational expense and time requirements increase
Solution Approach 1:
The patent replaces the expensive and time-consuming energy minimization and sampling procedures with a more efficient alternative: direct probabilistic sampling from the feedforward network's output distribution. Instead of performing iterative energy minimization to generate samples, the system uses the network's learned probability distribution to directly sample new data points, achieving similar generative results with significantly reduced computational cost and time
Data Source
AI summary
Recurrent neural networks are powerful tools for handling incomplete data problems in machine learning thanks to their significant generative capabilities. However, the computational demand for algorithms to work in real time applications requires specialized hardware and software solutions. We disclose a method for adding recurrent processing capabilities into a feedforward network without sacrificing much from computational efficiency. We assume a mixture model and generate samples of the last hidden layer according to the class decisions of the output layer, modify the hidden layer activity using the samples, and propagate to lower layers. For an incomplete data problem, the iterative procedure emulates feedforward-feedback loop, filling-in the missing hidden layer activity with meaningful representations.