Binary Last Memory for Sparse Temporal State Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learnable models, especially recurrent neural networks, face significant memory constraints when processing temporal data, limiting their applicability due to the need to store large internal states, which is particularly challenging in applications with insufficient memory resources.
Innovation Solution
A binary last memory system is introduced, where each element of the internal state stores the duration since the last occurrence of a binary state value, allowing for efficient storage of previous states by encoding the most recent occurrence rather than each individual state, thereby covering larger time horizons with the same memory footprint.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If previous states are stored in a temporal window for recurrent computation, then the machine learnable model can process temporal data effectively, but memory resources are consumed excessively
Solution Approach 1:
The patent extracts only the essential temporal information needed for recurrent computation by storing the last occurrence time of each state element rather than the complete temporal window. This selective extraction of critical temporal data reduces memory consumption while preserving the ability to reconstruct previous states when needed.
Solution Approach 2:
Instead of storing states in the conventional forward temporal sequence (x(t), x(t-1), x(t-2), ...), the patent inverts the approach by storing the backward-looking temporal information (last occurrence times). This inversion allows the system to reconstruct temporal sequences on-demand rather than maintaining them continuously in memory.
2Quantity of substance
If fewer previous states are stored to reduce memory usage, then memory resources are conserved, but the performance of the machine learnable model decreases
Solution Approach 1:
The patent performs preliminary computation by storing the last occurrence times of state elements during the forward pass. This preliminary action enables efficient reconstruction of temporal sequences when needed for loss computation or gradient calculation, ensuring model performance is maintained without requiring continuous storage of all previous states.
Solution Approach 2:
The patent changes the parameter representation from storing complete state vectors to storing scalar time indicators (last occurrence times). This parameter transformation reduces memory footprint from O(N*T) to O(N) while maintaining the ability to reconstruct temporal information when required for training and inference.
3Loss of information
If a temporal window of previous states is stored, then accurate temporal information is available for computation, but the memory footprint increases significantly
Solution Approach 1:
The patent extracts only the critical temporal marker (last occurrence time) for each state element rather than storing the entire temporal window. This extraction preserves sufficient temporal information for accurate reconstruction when needed while dramatically reducing the memory footprint from storing all intermediate states.
Solution Approach 2:
The patent creates a compact copy of temporal information in the form of last occurrence times, which can be used to reconstruct the full temporal sequence when required. This copying approach stores minimal information that serves as a key to regenerate the complete temporal context on-demand.
Data Source
AI summary
Systems and computer-implemented methods for training a machine learnable model and for using the machine learned model for inference, both of which using only limited memory resources. During training and inference, the machine learnable model uses previous state information. A state memory is provided which efficiently stores this previous state information. Instead of storing each previous state individually and integrally, for each element of the internal state, a value is stored in the state memory which is indicative of a most recent occurrence of an element of the internal state of the machine learnable model holding or transitioning to a particular binary state value. The states of the machine learnable model are representable as binary values and when states infrequently hold or transition to a particular binary state value.