Temporally Agnostic Transformer Training for Event Order Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transformer models struggle to understand timing contextually within data when generating predictions, particularly when dealing with sequences of events, as they are traditionally trained for sequences of text rather than sequences of events, which affects their ability to make accurate predictions and classify sequences.
Innovation Solution
Training transformer models to recognize perturbations in event sequences by switching the order of event pairs and optimizing parameters to be agnostic to time, using attention matrices and loss functions to enhance their understanding of event ordering and anomalies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If transformer models are trained to process sequences of text, then they can effectively handle grammatical and contextual structuring, but they fail to understand timing contextually when dealing with sequences of events
Solution Approach 1:
The patent applies local quality by introducing event-type specific processing within the transformer architecture. Different event types (e.g., user actions, system responses, errors) are handled with specialized attention mechanisms and embedding strategies, allowing the model to maintain general transformer capabilities while adapting to the specific temporal and contextual requirements of time series event data
2Device complexity
If transformer models process time series data with traditional text-based training, then they can maintain their attention matrix capabilities, but they cannot detect data anomalies or understand event ordering impacts
Solution Approach 1:
The patent applies preliminary action by pre-processing time series event data into structured formats before inputting to the transformer model. Event sequences are pre-tokenized, temporal relationships are pre-encoded, and anomaly indicators are pre-computed, allowing the transformer to focus on pattern recognition rather than basic data processing, thereby enhancing anomaly detection without significantly increasing model complexity
Solution Approach 2:
The patent introduces intermediary components between the raw time series data and the transformer model, including event embedding layers, temporal attention mechanisms, and loss function intermediaries. These intermediaries translate temporal sequences into representations that the transformer can process effectively, bridging the gap between traditional text processing capabilities and time series analysis requirements
3Productivity
If transformer models are used for event sequence prediction, then they can leverage their powerful processing capabilities, but they lack the ability to recognize perturbations in event sequences
Solution Approach 1:
The patent applies dynamics by implementing dynamic attention mechanisms that adaptively adjust their focus based on the input event sequence. The attention weights are dynamically computed to emphasize critical event orderings and temporal relationships, allowing the model to maintain high processing speed while becoming sensitive to perturbations in event sequences through learned temporal patterns
Data Source
AI summary
Methods and systems are described herein for training and implementing a temporally agnostic transformer model. To train the transformer model, event sequence data associated with users is modified by selecting pairs of events and switching an ordering of those events within the sequence. The modified event sequence data is provided to a transformer model, which produces an attention matrix indicating which pairs of events the model focused on when performing its predictions. Using a reference matrix, attention values associated with the switched pairs of events can be identified and the cross entropy can be maximized. This optimization can be used to update the transformer model to obtain a transformer model that is agnostic to event ordering.