Transformer Local Context for Scalable Tabular Classification

Solution Overview

Problem

Transformer-based models struggle to scale effectively to larger and more complex tabular datasets due to memory limitations and inefficient processing of global context, leading to decreased predictive performance.

Innovation Solution

The solution involves replacing global context with local context generated from tabular data using a transformer model, reducing the amount of context processed during inference and optimizing processor consumption and storage requirements.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If global context is used in transformer model for tabular data processing, then comprehensive information is available for prediction, but memory consumption and processing time increase significantly

Engineering Contradiction:

Improvepredictive performanceVSAvoidmemory consumption

Core Design Contradiction:

ReliabilityVSQuantity of substance

Solution Approach 1:

The patent segments the global context into multiple local contexts, each representing a specific region or aspect of the tabular data. Instead of processing the entire dataset as a single global context, the transformer model processes multiple smaller local contexts independently, then aggregates their results. This segmentation reduces the memory burden on each processing unit while maintaining comprehensive coverage through aggregation of multiple segments.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces a new dimension of processing by organizing context information not just as a flat global structure but as a hierarchical or spatially-organized set of local contexts. This dimensional reorganization allows the system to access and process context information more efficiently, transforming the problem from processing one large context to processing multiple smaller contexts in a structured arrangement.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

2Reliability

If global context is processed during inference, then complete data availability is achieved, but processing speed decreases

Engineering Contradiction:

Improveprediction accuracyVSAvoidprocessing speed

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The patent segments the global context into multiple local contexts, each representing a specific region or aspect of the tabular data. Instead of processing the entire dataset as a single global context, the transformer model processes multiple smaller local contexts independently, then aggregates their results. This segmentation reduces the memory burden on each processing unit while maintaining comprehensive coverage through aggregation of multiple segments.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent applies partial action by processing only the necessary local contexts required for each prediction task rather than always processing the complete global context. This selective processing approach maintains prediction accuracy for tasks that don't require all available information, thereby reducing processing time and resource consumption while still achieving sufficient predictive performance.

Inventive Principle:

Principle #16Partial or excessive action

3Quantity of substance

If transformer model memory size is limited, then resource efficiency is improved, but the amount of input data that can be processed is reduced

Engineering Contradiction:

Improvememory capacityVSAvoiddata processing capability

Core Design Contradiction:

Quantity of substanceVSAdaptability or versatility

Solution Approach 1:

The patent segments the global context into multiple local contexts, each representing a specific region or aspect of the tabular data. Instead of processing the entire dataset as a single global context, the transformer model processes multiple smaller local contexts independently, then aggregates their results. This segmentation reduces the memory burden on each processing unit while maintaining comprehensive coverage through aggregation of multiple segments.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent creates a universal processing framework where the same transformer model with limited memory can handle various tabular data processing tasks by selectively applying different local contexts. The system is designed to be multi-functional, adapting to different prediction tasks and data types using the same core architecture, thereby achieving versatility without requiring proportional increases in memory capacity.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS20250384288A1Contextual classification of tabular data for self-attention

Publication Date: 2025.12.18 THE TORONTO DOMINION BANK

AI summary

An example operation may include at least one of storing tabular data in a database, receiving an input sequence by a transformer model that includes global context, generating a query vector from the input sequence, wherein the query vector corresponds to a data record within the tabular data, generating local context comprising at least one additional vector from the input sequence within a proximity threshold to the query vector within vector space, replacing the global context of the transformer model with the local context, and generating an output based on execution of the transformer model with the local context on the query vector and the tabular data.