Recurrent Neural Network Online Learning with Fixed Reservoir Weights
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for machine learning of multi-dimensional time series data using recurrent neural networks face high calculation costs due to inverse matrix operations, making online learning difficult.
Innovation Solution
A machine learning device employing a recurrent neural network with fixed weights in the reservoir layer and a feed-forward readout layer, utilizing an extended Kalman filter method for weight updating, allows for efficient online learning of time series data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If inverse matrix operation is performed to update weights in recurrent neural network, then machine learning of multi-dimensional time series data can be achieved, but calculation cost increases
Solution Approach 1:
The patent extracts the weight updating mechanism from the traditional recurrent neural network and replaces it with a reservoir computing architecture. The reservoir layer maintains fixed random weights, eliminating the need for inverse matrix operations while preserving the ability to process multi-dimensional time series data through the readout layer.
Solution Approach 2:
The patent substitutes the mechanical inverse matrix operation with a simplified linear algebra operation in the readout layer. Instead of performing computationally expensive matrix inversion to update weights, the system uses matrix multiplication and addition operations that are computationally efficient while achieving the same learning objective.
2Adaptability or versatility
If inverse matrix operation is performed to update weights, then weight updating can be achieved, but online learning becomes difficult
Solution Approach 1:
The patent removes the weight updating process from the reservoir layer and relocates it to the readout layer. This extraction allows the reservoir to maintain fixed weights ideal for online processing, while the readout layer handles adaptive weight adjustments without requiring inverse matrix operations.
Solution Approach 2:
The patent pre-configures the reservoir layer with fixed random weights before processing begins. This preliminary action eliminates the need for repeated inverse matrix operations during online learning, as the reservoir's transformation capabilities are established in advance and only the readout layer weights require updating.
Data Source
AI summary
A machine learning device performing online learning of input data of one or more dimensions aligned in a pre-determined order using a recurrent neural network having a plurality of nodes connected by edges to which weights are assigned performs an output data generating process and a weight updating process every time the input layer receives the input data of one or more dimensions in the pre-determined order, in which, the weight updating process is a process in which a weight assigned to each edge connecting a 1st intermediate node and a 2nd intermediate node and a weight assigned to each edge connecting a 2nd intermediate node and an output node are updated using an equation derived based on an extended Kalman filter method, 1st intermediate data of one or more dimensions, and output data of one or more dimensions.