ML Accelerator Tile Scheduling for Wide-Column Data Locality
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine-learning accelerator scheduling techniques suffer from poor hardware utilization, weight reuse, and activation locality, leading to inefficient data reshuffling and increased system latency.
Innovation Solution
Implement topological scheduling, which partitions tile and activation columns into wide columns and binds them to improve hardware utilization, reduce data shuffling, and enhance activation locality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional scheduling techniques are used, then hardware utilization is poor, but implementing topological scheduling increases hardware utilization
Solution Approach 1:
The patent segments the tile array into multiple wide columns and partitions the activation rectangle into corresponding wide columns. This segmentation allows independent processing of different column groups, improving hardware utilization by enabling parallel execution across segmented units while maintaining manageable scheduling complexity through localized resource allocation.
Solution Approach 2:
The patent introduces a new dimensional organization by grouping tiles into wide columns rather than traditional single-column arrangements. This dimensional change in resource organization allows multiple activation columns to be processed simultaneously across the wide column structure, significantly improving hardware utilization without proportionally increasing scheduling complexity.
2Adaptability or versatility
If data is shuffled between layers, then flexibility is maintained, but system latency increases
Solution Approach 1:
The patent performs preliminary organization of activation data into wide column partitions that pre-align with the wide column structure of the tile array. This preliminary action ensures that data is already in the optimal position for processing, eliminating the need for post-processing shuffling operations and reducing system latency while maintaining data flexibility.
Solution Approach 2:
The patent enables continuous processing by maintaining data in a format that aligns with the wide column architecture throughout the computation pipeline. Data flows continuously through the system without interruption for reshuffling, as the initial wide column partitioning maintains compatibility with the processing units, thereby reducing latency while preserving adaptability.
3Ease of operation
If complex reshuffling is performed, then data locality is improved, but device complexity increases
Solution Approach 1:
The patent applies local quality by organizing data and resources into matching wide column structures, where each local unit (wide column in activation rectangle corresponds to wide column in tile array) is self-contained and optimally organized. This local optimization improves activation locality without requiring global reshuffling operations, thereby reducing device complexity while maintaining data locality.
4Speed
If more data conveyor bandwidth is allocated, then processing speed increases, but hardware resource consumption increases
Solution Approach 1:
The patent merges multiple activation columns into wide columns that are processed together by corresponding wide columns of tiles. This merging reduces the number of separate data conveyor channels needed, as data for multiple columns is transferred and processed in unified batches, thereby maintaining processing speed while reducing overall data conveyor bandwidth consumption.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for performing topological scheduling on a machine-learning accelerator having an array of tiles. One of the methods includes performing, at each time step of a plurality of time steps corresponding respectively to columns within each of a plurality of wide columns of the tile array, operations comprising: performing respective multiplications using tiles in a respective tile column for the time step, computing a respective output result for each respective tile column for the time step including computing a sum of results of the multiplications for the tile column, and storing the respective output result for the tile column in a particular output RAM having a location within the same tile column and on a row from which the output result will be read by a subsequent layer of the model.