Master Processing Unit Shareable Data Structure Memory Allocation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing compilation methods for graph-based programs on multi-tile processing systems lead to inefficient memory usage due to the need for duplicating shared data structures across multiple processing units, which restricts the size of models that can be executed on the chip and increases memory burden.
Innovation Solution
A method where shared read-only elements are identified and allocated to a master processing unit, allowing them to be transmitted and stored only on the processing units that need them, reducing the need for duplicate storage across all tiles and optimizing memory usage by using a shareable data structure approach.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If shared data structures are duplicated across all processing units, then each processing unit has access to the data it needs, but memory usage increases significantly
Solution Approach 1:
The patent merges the storage of shared data structures into a single location (master processing unit) rather than duplicating them across all processing units. The master processing unit stores the shared data structure once, and other processing units access it through coordinated memory operations, thereby combining multiple access needs into a single storage location.
Solution Approach 2:
The patent introduces an intermediary mechanism where the master processing unit acts as a mediator between processing units that need shared data. Instead of direct access to duplicated data, processing units request and receive data through the master processing unit, which manages the shared data structure and distributes it as needed.
2Quantity of substance
If shared data structures are stored only on the master processing unit, then memory usage is reduced, but access time increases due to inter-unit communication
Solution Approach 1:
The patent applies preliminary action by pre-loading shared data structures into the master processing unit's memory before other processing units need them. The compiler identifies and prepares shared data structures in advance, so when processing units need the data, it is already available in the master unit, reducing the actual access time during execution.
Solution Approach 2:
The patent introduces dynamic data sharing mechanisms where the master processing unit can dynamically transmit shared data structures to other processing units based on real-time needs. The system dynamically determines which processing units need which data and transmits only the necessary portions, optimizing access time based on actual usage patterns rather than static pre-distribution.
3Ease of manufacture
If data structures are allocated to each processing unit individually, then memory allocation is simple, but the system cannot execute larger models due to memory constraints
Solution Approach 1:
The patent makes the master processing unit's memory serve multiple functions: it stores both local data for the master unit and shared data structures for all other processing units. This multi-functional memory allocation allows the system to handle larger models by consolidating shared data in one location rather than requiring separate allocations across all units.
Solution Approach 2:
The patent changes the dimension of data storage from a distributed horizontal allocation across multiple processing units to a centralized vertical structure where the master processing unit serves as a central repository. This dimensional shift allows the system to accommodate larger models by utilizing the master unit's memory capacity as a shared resource for the entire system.
Data Source
AI summary
In a computer comprising multiple processing units, a method of exchanging read only elements between the processing units is described. The read only elements may be code or data, such as vector or matrix data for an AI graph. A master processing unit is identified. At compile time, at least one shareable read only element is allocated to the master processing unit. The at least one shareable read only element is stored in the local memory of the master processing unit. At compile time a transmitting exchange code sequence designated to be executed at the execution stage of the master processing unit is also allocated to the master processing unit. At a time point determined at compile time, the transmitting exchange code sequence causes the processing unit to identify the shareable read only element and to generate a message to be transmitted for reception by another processing unit, the message comprising the shareable read only data element.