Dynamic Memory Allocation for ML Accelerator Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In machine learning accelerator communications networks, the nondeterministic nature of data communications leads to variable latencies, causing inefficiencies in data transmission and processing.
Innovation Solution
A method for memory allocation in machine learning accelerator communications networks involves accessing metadata for communications ports, determining expected latency, and allocating portions of shared memory based on latency and traffic patterns, with dynamic reassignment of memory as needed during execution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If fixed memory allocation is used for communication ports, then device complexity is reduced, but communication efficiency deteriorates due to variable latencies
Solution Approach 1:
The patent implements dynamic memory allocation where the memory portion for each communication port is adjusted based on real-time latency measurements. The system continuously monitors actual latencies and reallocates memory buffers accordingly, transforming the static memory allocation into a dynamic adaptation mechanism that responds to changing network conditions.
Solution Approach 2:
The system changes the memory allocation parameter based on measured latency values. By using latency as a feedback parameter, the system adjusts the size of memory portions allocated to different ports, optimizing buffer sizes to match actual communication patterns and reducing waste from fixed allocation.
2Productivity
If memory is allocated based on expected latency, then communication efficiency is improved, but device complexity increases due to dynamic allocation requirements
Solution Approach 1:
The communication ports perform self-measurement of latencies and self-determination of their memory allocation requirements. Each port monitors its own communication performance and triggers reallocation when needed, reducing the need for complex centralized control while maintaining optimization.
Solution Approach 2:
The system implements a feedback loop where actual latency measurements are continuously monitored and used to adjust memory allocation. This closed-loop control ensures that memory portions are optimized based on real performance data, automatically adapting to changing communication patterns without manual intervention.
3Quantity of substance
If zero memory is assigned to unused ports, then memory utilization is optimized, but reliability may be affected if port usage changes
Solution Approach 1:
The system dynamically adjusts memory allocation in response to changes in port usage. When a port transitions from unused to active, the system detects the change in latency patterns and reallocates memory accordingly, ensuring that memory is available when needed while maintaining high utilization during stable periods.
Data Source
AI summary
This disclosure generally provides solutions for improving the performance of a custom-built, packet-switched, TPU accelerator-side communication network. Specifically a set of solutions to improve the flow-control behavior by tuning the packet buffer queues in the on-chip router in the distributed training supercomputer network are described.