Time-Series Transformer Forecasting With Space-Time Attention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing time-series forecasting models struggle with scalability and generalization across diverse datasets, particularly when handling high cardinality, high time resolution, sparsity, right skew, and outliers in infrastructure data, leading to poor performance.
Innovation Solution
A forecasting foundation model utilizing a factorized transformer architecture with space-time attention blocks and a probabilistic mixture model head, specifically a Student-T mixture model, to generate multivariate probabilistic predictions, allowing for efficient processing and handling of complex time-series data characteristics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If separate models are developed for each metric, then model accuracy for specific metrics can be optimized, but scalability is limited and maintenance complexity increases
Solution Approach 1:
The patent combines multiple metric-specific models into a single unified foundation model that can process and forecast diverse time-series metrics simultaneously. This consolidation reduces the number of separate models from many individual models to one general-purpose model, directly addressing the scalability and maintenance complexity issues while preserving forecasting accuracy through the model's ability to learn from multiple metrics during training
2Device complexity
If basic forecasting models are used, then model simplicity is maintained, but performance on diverse datasets with high cardinality, sparsity, and outliers is poor
Solution Approach 1:
The patent employs parameter changes by utilizing a Student-T mixture model head instead of standard prediction heads. The Student-T distribution parameters (location, scale, and degrees of freedom) allow the model to adapt to diverse data characteristics including outliers and heavy tails, improving reliability on complex time-series data while maintaining a relatively simple transformer-based architecture
3Adaptability or versatility
If foundation models are used to improve generalization, then adaptability across different metrics is enhanced, but computational overhead increases
Solution Approach 1:
The patent segments the processing of multivariate time-series data by dividing it into patches along the time dimension before feeding into the transformer. This patch-based segmentation reduces the computational burden on the foundation model by breaking down large sequences into manageable chunks, enabling efficient processing while maintaining the model's generalization capabilities across different metrics
4Measurement precision
If models are retrained and tuned for diverse datasets, then forecasting accuracy can be improved, but time and cost increase significantly
Solution Approach 1:
The patent creates a universal foundation model trained on multiple diverse metrics simultaneously, enabling it to generalize across different types of time-series data without requiring separate retraining for each metric. This multi-functional approach allows the single model to adapt to new metrics through inference alone, eliminating the time-consuming retraining and tuning process while maintaining high forecasting accuracy
Data Source
AI summary
The present disclosure describes technology for training and deploying time-series optimized transformers for observability (TOTO). The system may process multivariate time-series data using an artificial intelligence (AI) model. The model may include a patch embedding layer and a transformer architecture. The patch embedding layer is configured to receive the multivariate time-series data and output patch embeddings. The transformer architecture is configured to process the output patch embeddings and output transformed embeddings. The transformer architecture may include segments, with each segment including at least one space-wise block and a configurable number of time-wise blocks.