Multivariate Time Series Forecasting With Unified Attention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models for multivariate time series forecasting struggle to capture intricate cross-channel and cross-time dependencies effectively.
Innovation Solution
A transformer-based model with a unified attention mechanism is employed, flattening patches from different variates into a unified sequence to simultaneously model inter-variate and intra-variate dependencies, utilizing a dispatcher module to reduce complexity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If existing machine learning models are used for multivariate time series forecasting, then the forecasting task can be performed, but the model cannot capture intricate cross-channel and cross-time dependencies effectively
Solution Approach 1:
The patent segments the multivariate time series data into patches along the temporal dimension. Each patch captures local temporal patterns, and the segmentation enables the model to process intricate dependencies by breaking down the complex time series into manageable units that can be analyzed both within and across channels
Solution Approach 2:
The patent transforms the traditional time series representation by introducing a patch dimension. Instead of processing raw time steps directly, the model operates on patches that aggregate temporal information, thereby adding a new dimension for capturing dependencies while reducing the temporal complexity
2Measurement precision
If patches from different variates are flattened into a unified sequence to model inter-variate and intra-variate dependencies simultaneously, then forecasting accuracy improves, but computational complexity increases
Solution Approach 1:
The patent applies segmentation by dividing the flattened patch sequence into distinct variate groups. This allows the attention mechanism to operate efficiently by processing patches within and across variates in an organized manner, reducing the computational burden of modeling all dependencies simultaneously
Solution Approach 2:
The patent implements local quality by applying different attention mechanisms to different types of dependencies. Specifically, it uses separate attention operations for inter-variate dependencies (across different variates) and intra-variate dependencies (within the same variate), allowing each type of dependency to be modeled with appropriate computational resources
Data Source
AI summary
Embodiments described herein provide A method of training a neural network based model for predicting time series data. The method may include receiving, via a data interface, multi-variate time-series data; generating a plurality of tokens based on flattening the multi-variate time-series data; generating a first intermediate representation via a first cross-attention layer of the neural network based model with a plurality of dispatcher tokens as the query, and the plurality of tokens as the key and value; generating a second intermediate representation via a second cross-attention layer of the neural network based model with the plurality of tokens as the query, and the first intermediate representation as the key and value; generating a predicted time-series value based on the second intermediate representation; computing a loss based on a comparison of the predicted time-series value and a ground-truth value; and training the neural network based model based on the loss.