RNN Input Converting Unit for Dimensionality Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Recurrent neural networks (RNNs) face challenges in reducing computational complexity due to their increased complexity compared to non-recurrent networks, particularly in LSTM layers where gate outputs are calculated using both inputs and recurrent inputs, leading to inefficiencies in dimension allocation and reduced ability to automatically optimize dimension reductions.
Innovation Solution
The pattern recognition apparatus incorporates an input converting unit within RNN or LSTM layers that converts input and recurrent input vectors into smaller dimension vectors, allowing the RNN processor to calculate output vectors, thereby automatically optimizing dimension allocation and reducing computational complexity through linear or nonlinear transformations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If RNN layers use both input and recurrent inputs for calculation, then pattern recognition capability reflecting series patterns is improved, but computational complexity increases
Solution Approach 1:
The patent segments the input processing by introducing an input converting unit that separately processes the input vector and recurrent input vector before combining them. This segmentation allows for optimized dimensionality reduction of each component independently, reducing the overall computational complexity while maintaining the pattern recognition capability that relies on both input and recurrent inputs.
2Device complexity
If dimension of input and recurrent input vectors is reduced, then computational complexity is reduced, but ability to automatically optimize dimension reduction is reduced
Solution Approach 1:
The patent incorporates a feedback mechanism where the input converting unit uses the output from the RNN processor (recurrent input) to inform and optimize its own dimensionality reduction process. This feedback loop enables the system to automatically optimize the dimension reduction parameters based on actual processing needs, maintaining adaptability while achieving lower computational complexity through reduced vector dimensions.
Data Source
AI summary
According to an embodiment, a pattern recognition apparatus includes an RNN layer, as a middle layer, that includes an input converting unit and an RNN processor. The input converting unit performs conversion, for each step, on an input vector and a recurrent input vector, and calculates and outputs a converted vector of which the number of dimensions is smaller than the sum of the numbers of dimensions of respective the input vector and the recurrent input vector. The input vector is formed of a feature vector output from an input layer or an output of the RNN processor included in a lower RNN layer. The recurrent input vector is formed of an output of a previous step of the RNN processor. The RNN processor calculates an RNN output vector from the converted vector calculated in the input converting unit and outputs the RNN output vector, for each step.