Transformer Model Channel Independence for Time Series Forecasting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional transformer-based models for multivariate time series forecasting face high memory consumption and computational resource requirements, and they mix information across channels, reducing accuracy and lacking self-supervised learning capabilities.
Innovation Solution
The approach involves dividing input time series into univariate subseries, transforming them using a pre-trained transformer model, and concatenating the results for multivariate forecasting, while employing patching and channel independence to capture local information and reduce computational costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional transformer-based models process multivariate time series directly, then they can capture global patterns, but memory consumption and computational resource requirements increase significantly
Solution Approach 1:
The patent divides the multivariate time series into multiple univariate subseries, processing each channel separately through independent transformer models. This segmentation reduces the computational complexity and memory consumption by avoiding the need to process all variables simultaneously, while still capturing global patterns through the aggregation of individual channel predictions.
2Reliability
If conventional transformer-based models mix information across channels, then they can capture inter-channel relationships, but model accuracy decreases due to noisy information mixing
Solution Approach 1:
The patent separates the processing of different channels into independent univariate subseries, preventing noisy information mixing. Each channel is processed independently to preserve signal quality, while inter-channel relationships are captured through the architecture's ability to aggregate results from multiple independent processors, thus avoiding information degradation.
Solution Approach 2:
The patent applies different processing strategies to different channels by treating each univariate subseries independently with its own transformer model. This local quality approach allows each channel to be optimized individually, preventing the degradation that occurs when all channels are processed uniformly with noisy mixing, while still enabling global pattern recognition through result aggregation.
3Reliability
If conventional transformer-based models process complete multivariate sequences, then they can capture global patterns, but computational efficiency decreases
Solution Approach 1:
The patent segments the multivariate time series into univariate subseries that are processed independently and in parallel. This segmentation enables computational efficiency by reducing the sequence length and complexity for each processor, while still capturing global patterns through the aggregation of results from multiple independent transformer models, thus improving overall computational efficiency without sacrificing pattern recognition capability.
Data Source
AI summary
A method, system, and computer program product that is configured to: receive an input time series from an external device in a first system, divide the input time series to a set of univariate time subseries, transform the set of univariate time subseries into a univariate prediction result series using a transformer model, concatenate the univariate prediction result series to a multivariate predictive result, and output the multivariate predictive result for providing time series forecasting to a second system.