Causal Transformer Model for Temporal Sequence Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing predictive data analysis solutions face efficiency and reliability challenges in performing accurate predictive operations on temporal sequences, particularly in capturing the sporadicity and heterogeneity of sequence elements.
Innovation Solution
A causal transformer machine learning model is trained using temporal, structural, and positional embeddings to project training tokens into respective embedding spaces, enabling the generation of context-dependent representations and improving predictive accuracy by capturing the sporadicity and heterogeneity of sequence elements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional predictive data analysis methods are used, then the system is simpler to implement, but predictive accuracy deteriorates due to inability to capture sporadicity and heterogeneity of sequence elements
Solution Approach 1:
The patent segments the temporal sequence data into discrete tokens representing different elements (states, actions, outcomes, cumulative discounted future outcomes). Each token type is processed separately through embedding layers, allowing the model to capture specific characteristics of each element type while maintaining overall predictive accuracy.
Solution Approach 2:
The patent transforms 1D temporal sequence data into multi-dimensional embedding spaces. Training tokens are projected into embedding spaces that capture temporal, structural, and positional relationships, adding dimensional depth to the data representation and enabling more accurate prediction of future outcomes.
2Measurement precision
If complex embedding spaces and contextual representations are used, then predictive accuracy improves, but computational operations increase
Solution Approach 1:
The patent performs preliminary actions by pre-processing training tokens into embedding spaces before main prediction operations. Context-dependent representations are generated in advance during training, allowing the model to make faster predictions during inference without repeating complex computational steps.
Solution Approach 2:
The patent creates embedding space representations as copies of the original temporal sequence data. These embedded representations capture the essential characteristics of the data in a transformed format, allowing the model to work with compressed versions of the data that require fewer computational operations while maintaining predictive accuracy.
3Reliability
If comprehensive training data with multiple tuple types is used, then model reliability improves, but training data requirements increase
Solution Approach 1:
The patent creates a universal model structure that handles multiple tuple types (states, actions, outcomes, cumulative discounted future outcomes) through a unified embedding framework. The same model architecture and training process can process different tuple types, reducing the need for separate models or extensive data-specific tuning while maintaining reliability across diverse data types.
Data Source
AI summary
Various embodiments of the present disclosure provide methods, apparatus, systems, computing devices, computing entities, and/or the like for generating a prediction output comprising one or more actions by receiving data associated with encounters in a tuple form, tokenizing the encounters, training a causal transformer machine learning model configured to predict outcomes of actions by translating action tokens from the tokenized encounters into one or more embedding spaces, and training a causal transformer machine learning model to select the one or more actions based on embeddings from the one or more embedding spaces.