Die-to-Die Streaming Processor Packaging for Scalable ML Throughput
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional chip multiprocessor (CMP) architectures face challenges in scalability, performance, and usability due to increased hardware complexity, leading to scheduling difficulties and runtime stalls, especially with growing workloads in machine learning applications.
Innovation Solution
The proposed solution involves an integrated circuit with deterministic streaming processors (DSPs) featuring a functional slice architecture. Each DSP is divided into functional units organized into functional slices, allowing for efficient streaming of data and instructions across multiple dies in a die-to-die (D2D) dense packaging configuration. This configuration enables model-parallelism across multiple dies, optimizing data and instruction flow for improved performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If conventional CMP architectures are used with increased processing cores for machine learning workloads, then computational power is improved, but hardware complexity increases leading to scheduling difficulties and runtime stalls
Solution Approach 1:
The processor is divided into multiple functional slices (first functional slice, second functional slice, etc.), each handling specific operations. This segmentation allows independent operation of each slice, reducing scheduling complexity while maintaining high computational throughput for machine learning workloads.
Solution Approach 2:
The patent introduces a third dimension for data flow by stacking functional slices vertically across multiple dies (die-to-die packaging). This 3D architecture enables data to flow through multiple layers simultaneously, increasing computational power without proportionally increasing interconnect complexity.
2Productivity
If more ALUs and components are added to handle growing workloads, then processing capacity is improved, but scheduling difficulty increases
Solution Approach 1:
Processing capacity is increased by adding more functional slices rather than adding more components to existing slices. Each slice is a self-contained unit with its own data flow path, which maintains scheduling simplicity while collectively providing high processing capacity.
Solution Approach 2:
Each functional slice is designed to be multi-functional, capable of handling various machine learning operations (matrix multiplications, convolutions, etc.). This universality allows the system to scale processing capacity by replicating slices rather than designing specialized components for each operation.
3Ease of operation
If conventional on-chip network interconnection is used, then data exchange between cores is enabled, but data flow efficiency decreases for streaming operations
Solution Approach 1:
The patent transitions from 2D on-chip network interconnection to 3D vertical data flow through stacked functional slices. Data can flow directly between adjacent slices in the vertical dimension, providing shorter paths and higher efficiency for streaming operations compared to lateral network routing.
Solution Approach 2:
The patent introduces dedicated data flow paths and buffers as intermediaries between functional slices, enabling efficient data transfer without relying on the general-purpose on-chip network. These intermediaries optimize the data exchange for streaming workloads.
4Productivity
If architectural enhancements such as caches and branch predictors are added, then performance is improved, but worst-case performance remains unbounded
Solution Approach 1:
The deterministic processor uses segmented functional slices with explicit data flow paths, eliminating the need for caches and branch predictors. Each slice processes data in a predetermined sequence, ensuring that worst-case performance is bounded and predictable without requiring performance-enhancing architectural features.
Data Source
AI summary
Embodiments are directed to an integrated circuit with multiple dies connected in a die-to-die (D2D) configuration. The integrated circuit can include a first die and a second die connected to the first die via a D2D interface circuit in the D2D configuration forming a D2D structure with the first die. The D2D interface can connect a first plurality of superlanes of the first die with a second plurality of superlanes of the second die for streaming data between the first die and the second die along a first direction or a second direction orthogonal to the first direction.