Shared Memory Allocation for Neural Network Compute Tiles
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network processing systems face inefficiencies in performing inference computations due to the large number of instructions required for memory allocation, especially for large neural networks with multiple layers, leading to increased instruction bandwidth and memory usage, as well as potential errors in fetching separate instructions for iterative loops.
Innovation Solution
The system generates shared instructions that specify a common memory address and size for multiple computing units assigned to a neural network layer, allowing them to store and aggregate partial results efficiently, reducing the total number of instructions and memory usage by using a single instruction for multiple units.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If separate memory allocation instructions are issued for each computing unit, then each computing unit can independently manage its memory, but the total number of instructions increases significantly
Solution Approach 1:
The patent combines separate memory allocation instructions for multiple computing units into a single shared instruction. The controller issues one memory allocation instruction that is broadcast to multiple computing units, and each unit independently determines its specific memory address based on its identity, eliminating the need for multiple separate instructions while maintaining accurate memory management
Solution Approach 2:
The shared memory allocation instruction serves multiple computing units simultaneously, making the instruction universal. A single instruction format is designed to be applicable to N computing units, where each unit adapts the generic instruction to its specific needs by calculating its own offset address, reducing instruction bandwidth requirements by a factor of N
2Manufacturing precision
If separate instructions are fetched for each computing unit, then memory allocation can be precisely controlled, but instruction fetching time and downtime increase
Solution Approach 1:
The system performs preliminary configuration by embedding the computing unit's identity and memory offset calculation logic within the shared instruction. Each computing unit pre-calculates its specific memory address by combining the base address from the shared instruction with its own offset, eliminating the need for sequential instruction fetching and reducing downtime
3Ease of operation
If multiple separate instructions are used for memory allocation, then each computing unit has dedicated memory control, but system resource usage increases
Solution Approach 1:
The patent segments the memory address space into multiple regions, with each computing unit assigned a specific offset range. The shared instruction contains a base address, and each computing unit independently determines its segment by adding its identity-based offset to the base, maintaining dedicated memory control while using a single shared instruction structure
Solution Approach 2:
The system transitions from a one-dimensional approach (separate instructions for each unit) to a two-dimensional approach (single shared instruction with multi-dimensional address calculation). Computing units determine their specific addresses by combining the shared base address with their own offset dimension, reducing instruction memory usage while preserving independent memory control
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training giant neural networks. One of the methods includes obtaining data indicating a neural network comprising a plurality of layers; for each layer in a subset of the plurality of layers: assigning a subset of the plurality of computing units to at least partially perform inference computations associated with the layer; determining a memory size and a common memory address for the respective addressable memory unit of each computing unit assigned for the layer; and generating a shared instruction comprising a memory allocation instruction that, when executed by each of the subset of the plurality of computing units, causes the computing unit to store a result of performing inference computations associated with the layer in the determined common memory address with the determined memory size in the addressable memory of the computing unit.