Software-Defined Tensor Streaming Multiprocessors for Deterministic Scaling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High-performance computing systems face challenges in achieving both strong and weak scaling due to communication latency and bandwidth trade-offs, requiring novel chip architectures and scalable system architectures to efficiently handle tensor computations in machine learning applications.
Innovation Solution
A network of deterministic tensor streaming processors (TSPs) with a software-defined, high-radix interconnection network that enables deterministic communication and scheduling, utilizing hardware-aligned counters and a compiler to manage traffic across local and global links, ensuring synchronous operation and eliminating latency variance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If additional processing elements are deployed to increase throughput (weak scaling), then productivity is improved, but communication latency increases and bandwidth requirements worsen
Solution Approach 1:
The system segments processing elements into processing clusters, where each cluster contains multiple processing elements that can communicate with each other. This segmentation allows throughput to scale by adding clusters while maintaining low latency within each cluster through local communication paths.
Solution Approach 2:
The system introduces a hierarchical dimension to the network architecture, with intra-cluster links providing low-latency communication within clusters and inter-cluster links providing high-bandwidth communication between clusters. This dimensional hierarchy resolves the contradiction by allowing throughput scaling across clusters while maintaining latency performance within clusters.
2Loss of time
If additional processing elements are deployed for strong scaling, then execution time is reduced, but communication bandwidth requirements and system complexity worsen
Solution Approach 1:
The system segments the architecture into standardized processing clusters that can be independently designed and manufactured. This segmentation reduces system complexity by repeating proven cluster designs rather than designing monolithic systems, while still achieving execution time reduction through parallel processing across multiple clusters.
Solution Approach 2:
The system employs a nested hierarchical structure where processing elements are nested within processing clusters, which are nested within the global system. This nesting allows strong scaling by composing larger systems from smaller standardized units, reducing overall system complexity through modular design while maintaining low execution time through hierarchical parallelism.
3Productivity
If a scalable system architecture with high throughput is designed, then productivity is improved, but network diameter increases and end-to-end latency worsens
Solution Approach 1:
The system segments the network into multiple processing clusters with local interconnections, reducing the effective network diameter for most communications. Throughput scales by adding clusters, while latency is minimized by keeping frequently accessed data and computations within the same or adjacent clusters.
Solution Approach 2:
The system applies local quality by providing high-bandwidth, low-latency communication paths within processing clusters for local operations, while using standard inter-cluster links for remote operations. This differentiation allows throughput to scale system-wide while maintaining low latency for the majority of local communications.
Data Source
AI summary
A system contains a network of processors arranged in a plurality of nodes. Each node comprises a respective plurality of processors connected via local links, and different nodes are connected via global links. The processors of the network communicate with each other to establish a global counter for the network, enabling deterministic communication between the processors of the network. A compiler is configured to explicitly schedule communication traffic across the global and local links of the network of processors based upon the deterministic links between the processors, which enable software-scheduled networking with explicit send or receive instructions executed by functional units of the processors at specific times, to establish a specific ordering of operations performed by the network of processors. In some embodiments, the processors of the network of processors are tensor streaming processors (TSPs).