Neural Network Graph Transformation for Edge Device Memory Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep learning compilation methods, such as Symmetric Multi-Processing (SMP) graph, are inefficient due to their stage-based scheduling, which requires a whole buffer to store stage outputs and leads to time-consuming operations when performing inference on edge devices with limited memory resources.
Innovation Solution
An optimization-based auto graph transformation method and architecture that transforms the neural network graph into a pipeline structure with serially connected convolution layers and concatenation layers, allowing for partial output data caching in a ring buffer to reduce memory footprint and enhance cache utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If stage-based scheduling is used in SMP graph, then software tiles can be scheduled systematically, but a whole buffer is required to store stage output and stages must wait for previous stages to complete
Solution Approach 1:
The patent divides the computation graph into multiple independent pipelines, each processing a portion of the input data. This segmentation allows each pipeline to operate independently with its own smaller buffer, eliminating the need for a single large buffer while maintaining systematic scheduling through the modular pipeline structure.
Solution Approach 2:
The patent implements pipeline parallelism where intermediate results are produced and cached in ring buffers before all previous stages complete. This preliminary action allows subsequent stages to begin processing earlier, reducing wait time and enabling smaller buffer sizes while maintaining scheduling efficiency.
2Ease of manufacture
If stage-based scheduling is used in SMP graph, then software tiles can be scheduled systematically, but stages must wait until previous stages produce all data which is time consuming
Solution Approach 1:
The computation graph is segmented into multiple independent pipelines that can execute concurrently. Each pipeline processes a subset of data independently, allowing parallel execution and eliminating the sequential wait-time inherent in stage-based scheduling while preserving systematic organization through the pipeline architecture.
Solution Approach 2:
The patent enables continuous processing by implementing pipeline parallelism where multiple stages operate simultaneously on different data portions. This continuity eliminates idle wait time between stages, as each pipeline maintains continuous useful action through overlapping execution of different computation stages.
3Quantity of substance
If pipeline structure with partial output caching is used, then memory footprint is reduced and cache utilization is enhanced, but graph transformation complexity increases
Solution Approach 1:
The patent segments the computation graph into standardized pipeline units with consistent interfaces. This segmentation reduces transformation complexity by providing a modular, repeatable pattern for splitting and recombining operations, while enabling partial output caching in ring buffers to reduce memory footprint through efficient cache utilization.
Data Source
AI summary
The application discloses a compilation method, a data processing method and an apparatus thereof. Data representing a first graph characterizing the operations of a first neural network is obtained. The data representing the first graph is processed to transform the first graph into a second graph. A set of instructions for characterizing the second graph is generated. The set of instructions is provided to one or more hardware platforms.