Multi-scale Neural Network Time Series Forecasting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current time series forecasting methods, particularly those using Transformers, face challenges in handling long sequence time series due to computational drift and lack of model flexibility, despite advancements in attention mechanisms and module additions.
Innovation Solution
A multi-scale framework for time series forecasting that iteratively processes input data at different resolutions using an encoder-based model, applying down-sampling and up-sampling techniques to refine forecasts, while maintaining parameter and time complexity, and incorporating cross-scale normalization to mitigate distribution shifts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Length of moving object
If Transformers process all input sequence together using attention mechanism, then the model can process longer sequences, but computational drift away from desired solution occurs
Solution Approach 1:
The patent divides the long input sequence into multiple segments or chunks that are processed separately through the transformer model. Each segment is processed independently to maintain computational stability, and the results are then aggregated to form the final prediction. This segmentation approach prevents computational drift by avoiding processing the entire long sequence in a single pass, while still enabling the model to handle long-range dependencies through multiple localized attention operations.
2Adaptability or versatility
If RNN-based models process sequential inputs, then model flexibility is improved, but vanishing gradient problem occurs for long sequence prediction
Solution Approach 1:
The patent replaces the sequential processing mechanism of RNNs with the parallel attention mechanism of Transformers. Instead of using recurrent connections that process inputs sequentially and suffer from vanishing gradients, the model uses self-attention layers that can compute relationships between all sequence positions in parallel. This substitution maintains model flexibility for capturing temporal dependencies while eliminating the gradient stability issues inherent in RNN-based approaches.
3Productivity
If down-sampling is applied to reduce computational complexity, then processing speed improves, but forecast precision decreases
Solution Approach 1:
The patent employs a multi-scale processing strategy where the input sequence is processed at different resolution levels. Down-sampled versions of the sequence are used to capture long-term trends and patterns, while the original high-resolution sequence is used to capture fine-grained details. The model integrates information from multiple scales, allowing it to maintain forecast precision by leveraging both coarse-grained temporal patterns and fine-grained local variations, thus avoiding the precision loss that would result from single-scale down-sampling.
Data Source
AI summary
A method for operating a neural network using an encoder-based model to provide a time series forecast, the method comprising: down sampling a time series dataset to generate an initial input having a first scale resolution, such that the first scale resolution is less than a scale resolution of the time series dataset; processing as a first iteration, using the model, the initial input to generate a first output; upsampling by an upsampling function the first output to generate a second input having a second scale resolution, the second scale resolution being higher than the first scale resolution, such that the second input is based on the first output; and processing as a second iteration, using the model, the second input to generate a second output; wherein the second output represents a time series forecast of the time series dataset.