Cooperative Neural Networks for Temporal Memory
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural networks, such as Dynamic Boltzmann Machines, are limited in handling high-dimensional time-series data with nonlinear dynamics, as their memory is dependent on the number of visible units and maximum delay length, which restricts their ability to model complex temporal dependencies.
Innovation Solution
The integration of a recurrent neural network (RNN) with a Dynamic Boltzmann Machine (DyBM) enables longer temporal memory by using a nonlinear feature map and updating bias parameters based on RNN outputs, allowing the network to handle binary or real-valued data with nonlinear dynamics, without requiring backpropagation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Duration of action of stationary object
If the memory of DyBM is increased by adding more visible units or extending FIFO queue length, then temporal memory capacity is improved, but the model complexity and computational cost increase
Solution Approach 1:
The patent divides the memory function into two separate components: a recurrent neural network (RNN) that handles temporal memory through its hidden states, and a Dynamic Boltzmann Machine (DyBM) that processes current inputs. This segmentation allows the RNN to provide long-term memory without increasing DyBM's visible units or FIFO queue length, thus maintaining model complexity while improving temporal memory capacity.
Solution Approach 2:
The RNN acts as an intermediary between the input data and the DyBM. It processes historical information and transforms it into a compressed representation that feeds into the DyBM's bias parameters. This intermediary approach enables long temporal memory without directly increasing the DyBM's structural complexity.
2Ease of manufacture
If the DyBM uses a linear dynamical system with FIFO queues, then the implementation is simple, but it cannot capture nonlinear dynamics of high-dimensional time-series data
Solution Approach 1:
The patent merges the RNN, which excels at capturing nonlinear temporal dependencies, with the DyBM, which provides structured probabilistic modeling. This combination allows the system to handle both linear and nonlinear dynamics in high-dimensional time-series data, overcoming the limitations of pure linear FIFO-based approaches while maintaining the DyBM's implementation simplicity.
Solution Approach 2:
The patent changes the parameter update mechanism by using RNN outputs to dynamically adjust DyBM bias parameters. This allows the system to adapt to nonlinear patterns in the data through learned transformations rather than fixed linear transformations, enabling the DyBM to handle nonlinear dynamics while keeping the core structure simple.
Data Source
AI summary
Cooperative neural networks may be implemented by providing an input to a first neural network including a plurality of first parameters, and updating at least one first parameter based on an output from a recurrent neural network provided with the input, the recurrent neural network including a plurality of second parameters.