Distributing Model Data in Nodes to Reduce Remote Memory Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In electronic devices using multiple nodes to process input data through computational models, the distribution of model data among nodes leads to bandwidth consumption and processing overhead due to remote memory accesses, especially when model data is too large to be stored in individual nodes' memories, limiting the capacity of communication fabrics and node processing capabilities.
Innovation Solution
The model data is distributed among nodes based on predetermined conditions, such as frequency of access, with frequently accessed data being copied to local memories of other nodes, reducing the need for remote memory accesses and optimizing bandwidth usage by ensuring that model data is locally available where needed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If model data is distributed among multiple nodes using model parallelism, then the device can process larger models that exceed individual node memory capacity, but remote memory accesses increase bandwidth consumption and processing overhead
Solution Approach 1:
The system performs preliminary actions by identifying frequently accessed model data in advance and proactively copying it to local memories of multiple nodes before actual processing occurs. This anticipatory distribution reduces the need for remote memory accesses during execution, thereby lowering bandwidth consumption while maintaining the ability to handle large-scale models across distributed nodes
2Loss of energy
If full copies of model data are replicated in each node using data parallelism, then remote memory accesses are eliminated, but memory capacity requirements for each node increase significantly
Solution Approach 1:
Instead of uniformly replicating all model data across all nodes, the system applies local quality by distributing only the specific portions of model data that are frequently accessed to each node's local memory. This selective distribution optimizes the balance between local availability (reducing bandwidth consumption) and memory capacity utilization, allowing each node to store only what it needs for efficient processing
3Quantity of substance
If model data is stored in separate portions across multiple nodes, then individual node memory requirements are reduced, but processing speed decreases due to remote memory access overhead
Solution Approach 1:
The system performs preliminary identification and copying of frequently accessed model data to local memories before processing begins. This advance preparation ensures that hot data is locally available during execution, eliminating the need for slow remote memory accesses and thereby maintaining high processing speed while still utilizing distributed memory architecture
Solution Approach 2:
The system dynamically adapts the distribution strategy by identifying which model data portions are frequently accessed and adjusting the distribution accordingly. This dynamic approach optimizes processing speed by ensuring that the most accessed data is locally available, while still benefiting from the reduced memory capacity requirements of distributed storage
Data Source
AI summary
An electronic device includes a plurality of nodes, each node having a processor that performs operations for processing instances of input data through a model, a local memory that stores a separate portion of model data for the model, and a controller. The controller identifies model data that meets one or more predetermined conditions in the separate portion of the model data in the local memory in some or all of the nodes that is accessible by the processors when processing the instances of input data through the model. The controller then copies the model data that meets the one or more predetermined conditions from the separate portion of the model data in the local memory in the some or all of the nodes to local memories in other nodes. In this way, the controller distributes model data that meets the one or more predetermined conditions among the nodes, making the model data that meets the one or more predetermined conditions available to the nodes without performing remote memory accesses.