Type-Aware Transformer Embeddings for Static-Dynamic Sequential Data

Solution Overview

Problem

Transformer machine learning models struggle with resource inefficiency and difficulty in learning meaningful representations due to improper handling of static and dynamic data types in sequential datasets, leading to replication of static information and hindering model training.

Innovation Solution

A data type-aware transformer model is developed that separates static and dynamic fields, generating separate embeddings for each, and adjusts parameters based on dynamic field comparisons to improve training efficiency.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If static field data is processed together with dynamic field data in a standard transformer model, then the model can process complete interaction records, but static information is replicated upon every instance of data consuming memory space and processing power

Engineering Contradiction:

Improvemodel training efficiencyVSAvoidmemory space consumption

Core Design Contradiction:

ProductivityVSQuantity of substance

Solution Approach 1:

The patent segments the input data into static field data and dynamic field data, processing them through separate transformer models. The static field data is processed by a first transformer model to generate static interaction embedding representations, while dynamic field data is processed by a second transformer model to generate dynamic interaction embedding representations. This segmentation prevents replication of static information across all data instances, reducing memory consumption and improving training efficiency.

Inventive Principle:

Principle #1Segmentation

2Productivity

If static field data is processed together with dynamic field data in a standard transformer model, then the model can process complete interaction records, but static information consumes processing power and reduces difficulty during model training

Engineering Contradiction:

Improvemodel training efficiencyVSAvoidprocessing power consumption

Core Design Contradiction:

ProductivityVSUse of energy by stationary object

Solution Approach 1:

The patent segments the input data into static field data and dynamic field data, processing them through separate transformer models. The static field data is processed by a first transformer model to generate static interaction embedding representations, while dynamic field data is processed by a second transformer model to generate dynamic interaction embedding representations. This segmentation prevents replication of static information across all data instances, reducing memory consumption and improving training efficiency.

Inventive Principle:

Principle #1Segmentation

3Reliability

If static information is included in every data instance, then the model has access to complete information, but the static information prevents the transformer model from learning meaningful representations

Engineering Contradiction:

Improverepresentation learning qualityVSAvoidmodel architecture complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent segments the input data into static field data and dynamic field data, processing them through separate transformer models. The static field data is processed by a first transformer model to generate static interaction embedding representations, while dynamic field data is processed by a second transformer model to generate dynamic interaction embedding representations. This segmentation prevents replication of static information across all data instances, reducing memory consumption and improving training efficiency.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent merges the static interaction embedding representations and dynamic interaction embedding representations to form intermediate inputs, which are then processed by a third transformer model to generate final sequence embedding representations. This merging allows the model to access complete information from both static and dynamic fields while maintaining the benefits of separate processing, thereby improving representation learning quality without excessive complexity.

Inventive Principle:

Principle #5Merging (Combining)

Data Source

PatentUS20250259180A1Method, System, and Computer Program Product for Providing a Type Aware Transformer for Sequential Datasets

Publication Date: 2025.08.14 VISA INTERNATIONAL SERVICE ASSOCIATION

AI summary

Provided are methods that include receiving interaction data associated with a plurality of interactions, the interaction data including interaction records that include a plurality of fields including a static field and a dynamic field, generating a static interaction embedding representation based on static field data associated with the static field and a first transformer model, generating a plurality of dynamic interaction embedding representations based on dynamic field data associated with the dynamic field of a sequence of interaction records and a second transformer model, generating a first intermediate input and a plurality of second intermediate inputs, generating a static sequence embedding representation and dynamic sequence embedding representations based on a third transformer model, and generating at least one prediction based on inputting the static sequence embedding representation and the plurality of dynamic sequence embedding representations to a machine learning model. Systems and computer program products are also disclosed.