Online Time-Series Forecasting With Per-Layer Adaptive Memory
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep learning models for time series forecasting fail to adapt efficiently to the non-stationary nature of time series data due to batch-learning methods that randomly sample look-back and forecast windows, breaking the time-varying characteristics of the data.
Innovation Solution
The FSNet framework combines a deep neural network backbone with a per-layer adapter and an associative memory component to facilitate fast adaptation to both new and recurring patterns in time series data, using exponential moving averages to smooth gradients and sparse memory interactions for efficient learning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If batch-learning is used for training deep neural networks, then the model can be trained on large datasets, but the model becomes non-adaptive and fails to handle time-varying patterns in online forecasting
Solution Approach 1:
The patent implements a dual-network architecture where a slow learner (batch-trained model) provides stable baseline predictions while a fast learner (online adaptation module) dynamically adjusts to time-varying patterns. The fast learner continuously updates model parameters in an online setting, enabling the system to adapt to non-stationary data distributions while maintaining the stability of the slow learner's learned representations.
Solution Approach 2:
The patent introduces a knowledge distillation mechanism where the slow learner acts as a teacher model that guides the fast learner. The fast learner distills knowledge from the slow learner's predictions while adapting to new patterns, serving as an intermediary between the stable batch-trained model and the dynamic online forecasting requirements. This mediator approach allows gradual adaptation without complete retraining.
2Adaptability or versatility
If the model is frozen during evaluation, then computational resources are saved, but the model cannot adapt to concept drifts and non-stationary environments
Solution Approach 1:
The patent segments the learning process into two distinct components: a slow learner that performs comprehensive batch training periodically and a fast learner that handles rapid online adaptations. This segmentation allows the system to balance computational resource usage by performing intensive updates only when necessary (slow learner) while maintaining lightweight continuous adaptation (fast learner), thus adapting to concept drifts without constant heavy computation.
Solution Approach 2:
The patent dynamically changes model parameters through the fast learner's online update mechanism. When concept drift is detected or in non-stationary environments, the fast learner adjusts parameters in real-time based on incoming data distributions. This parameter change capability allows the model to adapt to concept drifts while the slow learner periodically stabilizes the parameter space, managing computational resources efficiently.
3Reliability
If random sampling of look-back and forecast windows is used during training, then data diversity is improved, but the time-varying nature of time series is broken
Solution Approach 1:
The patent implements dynamic window selection where the look-back and forecast windows are not randomly sampled but are adaptively chosen based on the detected patterns and characteristics of the time series data. The fast learner adjusts window parameters dynamically to preserve temporal dependencies and time-varying characteristics, ensuring that training samples reflect the actual temporal structure of the data rather than breaking it through random sampling.
Data Source
AI summary
Embodiments provide a framework combining fast and slow learning Networks (referred to as “FSNet”) to train deep neural forecasters on the fly for online time-series fore-casting. FSNet is built on a deep neural network backbone (slow learner) with two complementary components to facilitate fast adaptation to both new and recurrent concepts. To this end, FSNet employs a per-layer adapter to monitor each layer's contribution to the forecasting loss via its partial derivative. The adapter transforms each layer's weight and feature at each step based on its recent gradient, allowing a fine grain per-layer fast adaptation to optimize the current loss. In addition, FSNet employs a second and complementary associative memory component to store important, recurring patterns observed during training. The adapter interacts with the memory to store, update, and retrieve the previous transformations, facilitating fast learning of such patterns.