MAC Array Interleaved Striding for ML Bandwidth Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning hardware accelerators are bandwidth limited, leading to data traffic issues that hinder the performance of artificial neural networks, particularly in applications with high pixel counts or parameter-intensive networks.
Innovation Solution
A method involving a multiply-accumulate (MAC) array integrated with a memory architecture, utilizing interleaved striding and dilated convolutions, along with local partial sums and parallel access to memory interfaces, to efficiently process input data and neural network coefficients using a sliding window and prefetching techniques.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If machine learning hardware accelerators are used to improve processing performance, then computational speed is improved, but data bandwidth consumption increases causing bottlenecks
Solution Approach 1:
The patent implements prefetching of weight data into on-chip buffers before they are needed for computation. The weight data is loaded in advance from external memory into the MAC array's local memory, allowing the computational units to operate without waiting for data transfers during critical computational phases, thus reducing the effective data bandwidth requirement during execution.
Solution Approach 2:
The patent creates a hierarchical memory structure where external memory feeds into on-chip buffers, which then feed into the MAC array. This nested memory hierarchy allows the fastest (but smallest) on-chip buffers to hold frequently accessed weight data, while larger external memory provides the bulk storage, effectively managing bandwidth constraints at each level of the hierarchy.
2Productivity
If data is loaded into MAC array buffers, then computational efficiency is improved, but memory bandwidth consumption increases
Solution Approach 1:
The patent loads only the necessary weight data into the MAC array buffers rather than loading all possible data. By calculating the precise amount of weight data needed for the current computational task and loading only that portion, the system achieves high computational efficiency for the active workload while minimizing unnecessary memory bandwidth consumption that would occur with loading all data.
3Measurement precision
If weight data is fetched from external memory, then computational accuracy is maintained, but data transfer time increases
Solution Approach 1:
The patent performs weight data loading in advance of the computational task. By prefetching weight data into the MAC array's on-chip buffers before the computation starts, the system eliminates data transfer delays during the critical computational phase, maintaining both accuracy (through complete weight data availability) and speed (by avoiding transfer wait times).
Solution Approach 2:
The patent ensures continuous availability of weight data in the MAC array buffers by maintaining a prefetching mechanism that keeps the buffers supplied with new weight data as computational windows slide forward. This continuous supply eliminates gaps or pauses in computation caused by data transfer delays, maintaining uninterrupted useful computational action.
Data Source
AI summary
A method for accelerating machine learning on a computing device is described. The method includes partitioning neural network parameters and input data processed by a plurality of multiply-accumulate (MAC) units of a MAC array of the computing device. The method also includes interleaving MAC operations on the neural network parameters and the input data accessed according to a data sliding window and/or a stride N to compute an output during each cycle, in which N is greater than or equal to one.