Neural Network Layer Grouping for Edge Inference Memory Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network inference technologies face inefficiencies in energy consumption and latency, particularly when handling MobileNet-like architectures, due to high external memory access requirements, which limits their performance on edge devices.
Innovation Solution
A hardware-specific division method that groups neural network layers based on estimated duration and energy consumption, allowing for sequential inference processing within these groups, reducing external memory access and optimizing computation density and resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by stationary object
If neural network inference is performed layer by layer with frequent external memory access, then memory capacity is sufficient, but energy consumption increases and latency increases
Solution Approach 1:
The patent divides the neural network layers into multiple groups, where each group can be processed continuously using data already loaded into on-chip memory. This segmentation allows the system to process multiple layers without repeatedly accessing external memory, thereby reducing both energy consumption and latency associated with frequent memory accesses.
Solution Approach 2:
The patent loads activation data for multiple layers into on-chip memory in advance before processing begins. By preparing the data beforehand and keeping it in fast on-chip memory, the system avoids the need to access slower external memory during the actual inference computation, thus reducing latency and energy consumption.
2Productivity
If layers are divided into groups for sequential processing, then computation density increases, but device complexity increases
Solution Approach 1:
The patent employs a dynamic layer grouping strategy that adapts to different neural network architectures and hardware configurations. The system can adjust the number and composition of layer groups based on the specific computational requirements and available on-chip memory capacity, allowing flexible optimization without requiring complex fixed hardware structures.
Solution Approach 2:
The patent designs a universal layer grouping framework that can accommodate various neural network architectures (including MobileNet-like structures with depth-wise convolutions) and different hardware configurations. The same basic hardware architecture can handle different network types by adjusting the grouping parameters, reducing the need for specialized complex hardware for each architecture type.
3Loss of energy
If on-chip memory is used to store activation data, then external memory access decreases, but on-chip memory capacity requirements increase
Solution Approach 1:
The patent segments the neural network into layers that can be processed in groups, where each group's activation data fits within the available on-chip memory capacity. This segmentation allows the system to keep multiple layers' activation data in on-chip memory simultaneously, reducing external memory access while working within memory capacity constraints.
Solution Approach 2:
The patent dynamically adjusts the grouping parameters (number of layers per group, data precision, etc.) to optimize the balance between on-chip memory usage and external memory access frequency. By changing these parameters, the system can reduce energy loss from external memory access while respecting the physical constraints of on-chip memory capacity.
Data Source
AI summary
Neural network accelerator hardware-specific division of inference may be performed by operations including obtaining a computational graph and a hardware chip configuration. The operations also include dividing inference of the plurality of layers into a plurality of groups. Each group includes a number of sequential layers based on an estimate of duration and energy consumption by the hardware chip to perform inference of the neural network by performing the mathematical operations on activation data, sequentially by layer, of corresponding portions of layers of each group. The operations further include generating instructions for the hardware chip to perform inference of the neural network, sequentially by group, of the plurality of groups.