Reservoir Computing Model Size Reduction via Temporal State Reuse
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Reservoir computing requires a larger model size to achieve comparable performance to other machine learning models, leading to low calculation speed and power efficiency, and increased circuit size during hardware implementation.
Innovation Solution
A machine learning device and method that includes input, intermediate calculation, weighting, and learning means, where the intermediate calculation is performed multiple times, and the output is weighted and learned, allowing for high learning performance without increasing model size by reusing past states and outputs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If reservoir computing learns only the weight of an output layer, then high-speed learning is realized, but a larger model size is required to achieve the same performance
Solution Approach 1:
The patent applies preliminary action by pre-calculating and storing intermediate results (state quantities) at multiple time points before final output generation. The learning device stores state quantities from intermediate calculation results at different time points, and these pre-computed values are reused during output generation, eliminating the need for repeated full-scale calculations and reducing the model size required for achieving the same performance.
Solution Approach 2:
The patent introduces a time dimension by performing intermediate calculations at multiple time points (t-1, t-2, ..., t-n) and storing state quantities across these temporal dimensions. This transforms the problem from a spatial expansion (increasing model size) to a temporal exploitation (using historical state quantities), allowing the system to achieve comparable performance with a smaller model by leveraging information across time.
2Reliability
If the model size is large, then learning performance can be maintained, but calculation speed and power efficiency become low
Solution Approach 1:
The patent extracts and stores only the essential intermediate state quantities at multiple time points rather than maintaining a large complete model. By identifying and retaining only the necessary state variables from intermediate calculations that are needed for final output computation, the system achieves comparable learning performance with a smaller computational footprint, thereby improving calculation speed and power efficiency.
Solution Approach 2:
The patent discards redundant calculation paths by not performing full intermediate calculations during output generation, instead recovering and reusing previously stored state quantities from intermediate results. This selective discarding of unnecessary computations and recovery of essential stored values significantly improves calculation speed while maintaining learning performance.
3Reliability
If the model size is large, then performance can be maintained, but circuit size increases during hardware implementation
Solution Approach 1:
The patent applies preliminary action in hardware implementation by pre-computing and storing intermediate state quantities during the training phase, then reusing these stored values during inference. This eliminates the need for large circuits that would otherwise be required to perform repeated full-scale intermediate calculations, thereby reducing hardware footprint while maintaining performance.
Solution Approach 2:
The patent uses copying by storing replicated state quantities from intermediate calculations at multiple time points. Instead of implementing a large circuit that processes all possible intermediate values, the system creates and stores copies of essential state quantities that can be reused during output generation, significantly reducing the required circuit size while preserving performance.
Data Source
AI summary
A machine learning device includes: an input unit that acquires input data; an intermediate calculation unit that performs calculation on the input data a plurality of times; a weighting unit that performs weighting on an output of the intermediate calculation means for each of the plurality of times; an output unit that outputs output data based on a result of the weighting by the weighting means; and a learning unit that performs learning of a weight obtained by the weighting by the weighting means.