Transformer Attention Decomposition for Time-Aware Request Classification

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Transformer models struggle to understand and account for relative timing within data, particularly in time series data, leading to suboptimal performance in tasks involving authorization requests due to the influence of temporal components and limited context windows.

Innovation Solution

Decompose attention scores into event components and temporal components, adjusting the transformer model to reduce the impact of temporal components on attention scores based on time differences between events.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If transformer models process time series data using standard attention mechanisms, then the model can capture dependencies within sequences, but the model fails to understand relative timing between events leading to suboptimal performance

Engineering Contradiction:

Improveperformance accuracyVSAvoidtemporal understanding capability

Core Design Contradiction:

ReliabilityVSDifficulty of detecting and measuring

Solution Approach 1:

The attention score is segmented into two independent components: an event component capturing the importance of events and a temporal component capturing time differences. This segmentation allows the model to separately learn and optimize each aspect, resolving the contradiction between capturing event dependencies and understanding temporal relationships.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

A decomposed attention mechanism is introduced as an intermediary between the standard transformer architecture and the time series data. This intermediary layer explicitly models temporal relationships through time difference calculations, enabling the model to understand timing without sacrificing event dependency capture.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Productivity

If the transformer model uses fixed-length sequences to process time series data, then the model can operate efficiently, but temporal information is lost or memory usage becomes inefficient

Engineering Contradiction:

Improveprocessing efficiencyVSAvoidtemporal information

Core Design Contradiction:

ProductivityVSLoss of information

Solution Approach 1:

The model transitions from static fixed-length sequence processing to dynamic variable-length sequence processing. By calculating actual time differences between events rather than relying on fixed positional encodings, the model adapts to the temporal structure of the data, preserving temporal information while maintaining processing efficiency through the decomposed attention mechanism.

Inventive Principle:

Principle #15Dynamics

3Ease of operation

If the transformer model treats all positions in the sequence equally, then the model simplifies processing, but the model cannot capture the sequential nature of temporal data

Engineering Contradiction:

Improveprocessing simplicityVSAvoidtemporal dependency capture

Core Design Contradiction:

Ease of operationVSReliability

Solution Approach 1:

The decomposed attention mechanism applies different qualities to different components of the attention score. The event component weights events based on their intrinsic importance, while the temporal component weights them based on their time distance from the query event. This local differentiation allows the model to capture temporal dependencies while maintaining a relatively simple processing framework.

Inventive Principle:

Principle #3Local quality

4Device complexity

If the transformer model uses standard attention mechanisms without decomposition, then the model structure remains simple, but the model cannot distinguish whether high attention scores are due to event importance or temporal proximity

Engineering Contradiction:

Improvemodel architecture complexityVSAvoidattention score interpretation

Core Design Contradiction:

Device complexityVSMeasurement precision

Solution Approach 1:

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The model incorporates feedback mechanisms where the decomposed attention components are used to update and refine the model's understanding of temporal relationships. By analyzing whether high attention scores are driven by event importance or temporal proximity, the model can adjust its parameters to improve measurement precision without excessive architectural complexity.

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS12554810B2Decomposing attention values to de-emphasize temporal impacts of transformer model updating

Publication Date: 2026.02.17 CAPITAL ONE SERVICES LLC

AI summary

Methods and systems are described herein for decomposing attention values into event components and time components. The attention values may be generated by a transformer model based on time series data. The decomposed attention values can be used to identify instances where the transformer model classified a request into a first class based on an attention value that is influenced by its respective time component more than a threshold amount. In some embodiments, one or more modifications to those attention values may be applied to the influence of the time component on the attention values. The transformer model may reclassify the request based on the modified attention values.