Parallel AI Inference via Segmented Systolic and Self-Attention Circuits
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large systolic arrays are inefficient for AI operations that require data from previous tokens, such as self-attention operations, due to memory and processing limitations, while they excel in operations like matrix multiplication.
Innovation Solution
An integrated circuit (IC) with a systolic array for operations not using previous data sequences and a self-attention circuit for operations using previous data sequences, allowing parallel execution to improve throughput by processing independent tokens simultaneously.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a large systolic array is used for AI operations, then operations like matrix multiplication can be performed very efficiently, but operations requiring data from previous tokens (such as self-attention) cannot be performed efficiently due to memory and processing limitations
Solution Approach 1:
The AI processing system is segmented into two distinct circuits: a systolic array for operations not requiring previous token data (e.g., matrix multiplication), and a self-attention circuit for operations requiring previous token data. This segmentation allows each circuit to be optimized for its specific function, resolving the contradiction between high throughput for certain operations and adaptability for others.
Solution Approach 2:
The integrated circuit achieves multi-functionality by combining two specialized circuits that can handle different types of AI operations. The systolic array handles compute-intensive operations while the self-attention circuit handles operations requiring contextual memory, together providing universal processing capability for various AI model operations.
2Reliability
If a systolic array is used for self-attention operations, then memory and processing limitations are exceeded, but if not used, then operational efficiency is reduced
Solution Approach 1:
The self-attention functionality is extracted from the systolic array and implemented as a separate dedicated circuit. This extraction allows the systolic array to focus on operations it performs efficiently (matrix multiplications) while the self-attention circuit handles operations requiring contextual memory, preventing memory and processing limitations from being exceeded.
3Productivity
If parallel execution is implemented, then throughput is improved by processing independent tokens simultaneously, but device complexity increases
Solution Approach 1:
Two specialized circuits (systolic array and self-attention circuit) are merged into a single integrated circuit device. This merging enables parallel execution of different operation types, improving throughput while containing the complexity increase within a unified hardware architecture rather than requiring separate systems.
Data Source
AI summary
Embodiments herein describe an artificial intelligence (AI) hardware platform that includes at least one integrated circuit (IC) with a systolic array and a self-attention circuit. In one example, the systolic array performs operations in a layer of an AI model that do not use data from previous tokens or data sequences processed by the IC, while the self-attention circuit performs operations in the layer of the AI model that do use data from previous tokens or data sequences processed by the IC.