ML Inference Memory Management via Tile Grouping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The limited memory resources in ML-specific hardware, such as accelerators and dedicated hardware, struggle to accommodate the increasing size of activation tensors in machine learning models, leading to performance issues like higher latency and lower throughput.
Innovation Solution
The proposed solution involves a memory management system that links memory spaces across processing tiles with the same dimension for input and output tensors, creating grouped memory spaces to optimize memory usage. This system also prioritizes the removal of non-grouped memory spaces and moving data to secondary memory components to free up space.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored on the accelerator's local memory, then bandwidth and latency are improved, but the limited memory resources cannot accommodate growing data sets
Solution Approach 1:
The patent implements a nested memory hierarchy where local memory on processing tiles is nested within the broader memory system that includes secondary memory components. Activation tensors are stored in local memory when possible, and when local memory is full, the system nests additional storage capacity in secondary memory, creating a hierarchical structure that provides both fast local access and expanded total capacity.
Solution Approach 2:
The patent adds a new dimension to the memory architecture by introducing secondary memory components that expand the storage capacity beyond the single-dimensional local memory limit. This creates a multi-layered memory system where data can be distributed across different memory dimensions (local vs. secondary), effectively increasing total available memory capacity while maintaining fast access paths.
2Quantity of substance
If more memory resources are allocated to accommodate larger data sets, then memory capacity increases, but the cost and complexity of the hardware increases
Solution Approach 1:
The patent segments the memory system into distinct components: local memory on each processing tile and separate secondary memory components. This segmentation allows the system to achieve large total memory capacity without requiring a single large complex memory block, instead using multiple smaller, simpler memory units that can be independently managed and accessed.
Solution Approach 2:
The patent introduces memory management logic as an intermediary that coordinates between processing tiles and secondary memory components. This intermediary manages data placement, movement, and access coordination, allowing the system to achieve large effective memory capacity through software-controlled memory virtualization rather than requiring proportionally large physical memory hardware.
3Productivity
If data is moved between memory locations to optimize access patterns, then performance is improved, but additional time and energy are consumed
Solution Approach 1:
The patent performs preliminary actions by pre-loading activation tensors into local memory before they are needed for processing. The memory management system anticipates future memory needs and proactively transfers data from secondary memory to local memory in advance, so that when processing occurs, the data is already in the fast local memory, minimizing actual data movement time during critical processing paths.
Data Source
AI summary
A method includes receiving a machine learning (ML) network model in high-level code; generating an internal representation (IR), the IR is mapped to components in a multi-processing tile device; determining whether a first processing tile with a first on-chip memory (OCM) has a same dimension for an input/output tensor data as a second processing tile with a second OCM performing a same primitive function based on the IR; allocating a same memory address range within the first and the second OCM for the same primitive function if the first processing tile has the same dimension for the input/output tensor data as the second processing tile for the same primitive function; linking the memory address range of the first OCM to the memory address range of the second OCM to form a grouped memory space within the first and the second OCM respectively; and compiling low-level instructions based on the linking.