Transformer Token Selection for Lower Inference Compute

Solution Overview

Problem

Transformer neural networks are computationally inefficient due to processing all tokens equally, wasting resources on less relevant data, especially in tasks like object detection where static content is processed alongside moving objects.

Innovation Solution

Implement a predictive model to identify and prioritize relevant tokens for subsequent layers, bypassing less important tokens to reduce computational load while maintaining accuracy.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If all tokens are processed equally by transformer neural networks, then inference accuracy is maintained, but computational expense and memory usage increase significantly

Engineering Contradiction:

Improveinference accuracyVSAvoidcomputational expense

Core Design Contradiction:

ReliabilityVSUse of energy by moving object

Solution Approach 1:

The patent segments tokens into two distinct subsets based on their relevance to the second attention layer: a first subset of relevant tokens and a second subset of less relevant tokens. This segmentation enables selective processing where only the first subset is processed by the second attention layer, while the second subset is processed by a third attention layer, thereby reducing overall computational expense while maintaining inference accuracy.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent applies local quality by differentiating the processing quality for different token subsets. The first attention layer processes all tokens with full attention, while the second attention layer processes only the first subset of tokens with high relevance, and the third attention layer processes the second subset with lower relevance. This localized quality differentiation optimizes computational resources based on token importance.

Inventive Principle:

Principle #3Local quality

2Reliability

If all tokens are processed equally by transformer neural networks, then inference accuracy is maintained, but memory usage increases significantly

Engineering Contradiction:

Improveinference accuracyVSAvoidmemory usage

Core Design Contradiction:

ReliabilityVSQuantity of substance

Solution Approach 1:

The patent segments tokens into two distinct subsets based on their relevance to the second attention layer: a first subset of relevant tokens and a second subset of less relevant tokens. This segmentation enables selective processing where only the first subset is processed by the second attention layer, while the second subset is processed by a third attention layer, thereby reducing overall memory usage while maintaining inference accuracy.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent applies local quality by differentiating the processing quality for different token subsets. The first attention layer processes all tokens with full attention, while the second attention layer processes only the first subset of tokens with high relevance, and the third attention layer processes the second subset with lower relevance. This localized quality differentiation optimizes memory resources based on token importance.

Inventive Principle:

Principle #3Local quality

3Productivity

If a token prediction model is introduced to identify relevant tokens, then computational efficiency improves, but device complexity increases

Engineering Contradiction:

Improvecomputational efficiencyVSAvoidmodel architecture complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent introduces a token prediction model as an intermediary component between the first attention layer and the second attention layer. This intermediary model predicts the relevance of tokens to the second attention layer and generates predictions that are used to identify the first subset of relevant tokens. While this increases device complexity, it enables significant computational efficiency improvements by allowing selective processing of only relevant tokens.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS20250322275A1Token selection in transformer neural networks for efficient inferencing

Publication Date: 2025.10.16 QUALCOMM INC

AI summary

Certain aspects of the present disclosure provide techniques and apparatus for processing data using a transformer neural network. The method generally includes generating, via a first attention layer of a machine learning model, a first attention map based on an input into the machine learning model; identifying, using a token prediction model, a first subset of tokens in the first attention map more relevant to a second attention layer of the machine learning model and a second subset of tokens in the first attention map less relevant to the second attention layer of the machine learning model; generating, via the second attention layer of the machine learning model, a second attention map based on the first subset of tokens in the first attention map; and generating an inference based on the second attention map and the second subset of tokens in the first attention map.