MLA Compiler Scheduling Partial Networks for Memory Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning networks face challenges in efficient memory usage and data transfer due to the large volume of data and computations required, particularly when implemented on semiconductor dies with limited memory capacity and reduced power consumption, necessitating innovative methods to minimize temporary data storage and data transfers.
Innovation Solution
A machine learning accelerator (MLA) system with a compiler that intelligently orders computations to limit memory usage and data transfers by statically scheduling instructions, allocating computations to Tiles, and optimizing data transfer paths, allowing for efficient implementation of machine learning networks on edge devices with limited resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If machine learning networks are implemented on cloud-based compute facilities with significant resources, then computation performance is improved, but data transfer time and energy consumption increase
Solution Approach 1:
The patent segments the machine learning network computations into multiple independent stages that can be executed in different orders. By dividing the computation graph into separable operations, the system can optimize data flow and reduce the need to transfer entire datasets between cloud and edge, thereby reducing data transfer time while maintaining computation performance.
Solution Approach 2:
The patent applies preliminary actions by pre-processing and reordering computations before execution. The system analyzes the computation graph in advance to identify optimal execution sequences that minimize data transfer requirements, allowing edge devices to execute computations with locally available data rather than requiring continuous cloud communication.
2Loss of time
If machine learning networks are implemented on edge devices with limited memory, then data transfer is reduced, but memory capacity constraints increase
Solution Approach 1:
The patent applies dynamics by making the computation execution order flexible and adaptive. Instead of following a fixed sequential order, the system dynamically reorders operations based on data availability and memory constraints at runtime, allowing edge devices to efficiently utilize limited memory by loading only necessary data subsets when needed.
Solution Approach 2:
The patent changes parameters by modifying the execution sequence and data flow patterns. By altering the order in which computational operations are performed, the system can ensure that data remains in local memory longer and reduces the frequency and volume of data transfers, effectively working around limited memory capacity constraints.
3Quantity of substance
If computations are reordered to limit memory usage, then memory efficiency is improved, but computation complexity increases
Solution Approach 1:
The patent implements feedback mechanisms where the system continuously monitors memory usage and data availability during computation execution. Based on this feedback, the computation order is adjusted dynamically to maintain optimal memory efficiency. This feedback loop allows the system to manage computation complexity by making localized adjustments rather than requiring complete reordering of all operations.
Data Source
AI summary
A compiler manages memory usage in the machine learning accelerator by intelligently ordering computations of a machine learning network. The compiler identifies partial networks of the machine learning network representing portions of the machine learning network across multiple layers on which an output or set of outputs are dependent. Because any given output may depend on only a limited subset of intermediate outputs from the prior layers, each partial network may include only a small fraction of the intermediate outputs from each layer. Instead of implementing the MLN by computing one layer at a time, the compiler schedules instructions to sequentially implement partial networks. As each layer of a partial network is completed, the intermediate outputs can be released from memory. The described technique enables intermediate outputs to be directly streamed between processing elements of the machine learning accelerator without requiring large transfers to and from external memory.