Tensor Streaming Processor for Low Latency Transformer Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing processor-based systems experience latency issues when deploying transformer-based large language models, hindering real-time operation and requiring systems that can support low latency operations for these models.
Innovation Solution
The implementation of a Tensor Streaming Processor (TSP) architecture that performs low latency data processing by utilizing a streaming process model with interconnected computational and data storage tiles, enabling deterministic processing and parallel execution of transformer models like BERT through Matrix Multiplication and Vector Calculation modules, and optimizing operations such as softmax and layer normalization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If transformer models are deployed on traditional GPU/CPU systems, then the models can perform necessary calculations, but the systems experience greatly increased execution time and latency issues that prevent real-time use
Solution Approach 1:
The patent segments the transformer model computations into distinct operational phases (embedding layer, encoder layers with self-attention mechanisms, feed-forward layers) and maps them to specialized processing units. The TSP architecture divides the computational graph into discrete operations that can be executed in a deterministic pipeline, with each operation assigned to specific processing tiles. This segmentation enables precise control over execution timing and eliminates the variable latency characteristic of traditional GPU/CPU systems.
Solution Approach 2:
The patent introduces an intermediary layer between the model definition and hardware execution through the use of an intermediate representation (IR) and a compiler that translates high-level model definitions into deterministic TSP operations. This intermediary compilation layer transforms the computational graph into a sequence of deterministic operations that can be scheduled and executed with guaranteed timing, bridging the gap between flexible model definitions and rigid real-time execution requirements.
2Productivity
If traditional processor architectures are used for transformer models, then the models can be deployed, but the architectures require greatly increased time to execute, hindering real-time use
Solution Approach 1:
The patent replaces the traditional von Neumann architecture with a dataflow-based TSP architecture where computations are triggered by data availability rather than sequential instruction execution. Instead of using general-purpose CPU/GPU instruction pipelines, the system uses a compiler-generated dataflow graph that directly maps to hardware tiles, eliminating instruction fetch, decode, and pipeline stall overhead. This substitution of the underlying execution mechanism enables deterministic, high-speed execution of transformer operations.
Solution Approach 2:
The patent changes the execution model parameters from variable-latency instruction-based execution to fixed-latency operation-based execution. By transforming the computational graph into a sequence of deterministic operations with known execution times, the system achieves predictable performance. The compiler assigns specific operations to specific hardware tiles with guaranteed timing, changing the fundamental parameters of how computations are scheduled and executed to eliminate timing variability.
Data Source
AI summary
Facilitating efficient processing of transformer based models is provided herein. A low latency processing system includes a transformer having an embedding layer and a Tensor Streaming Processor (TSP) having a Matrix Multiplication module (MXM) and Vector Calculation module (VXM). The TSP is arranged to deterministically process information arranged by the embedding layer and an encoder layer with the associated self-attention mechanism, the information being further modified according to the transformer using a general matrix multiply (GEMM) mapped directly on the MXM and associated accumulator. Further, at least some set of information is processed to parallelize the execution of GEMMs across all MXM planes.