Depth-First DCN Inference with On-Chip Memory Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep convolutional neural networks (DCNs) face performance constraints due to limited resources, such as memory and processing capabilities, in embedded IoT devices, leading to increased off-chip memory access and energy consumption.
Innovation Solution
Implement depth-first processing for multi-layer artificial neural networks, determining partitions based on on-chip memory usage and off-chip memory write-back size to reduce off-chip memory access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If depth-first processing with partitioning is implemented, then off-chip memory access is reduced and energy consumption decreases, but device complexity increases due to partition determination logic
Solution Approach 1:
The neural network is divided into multiple partitions, where each partition processes a subset of layers. This segmentation allows intermediate results to be stored in on-chip memory buffers, reducing the need for frequent off-chip memory accesses and thereby lowering energy consumption while managing device complexity through structured organization
Solution Approach 2:
The system performs preliminary determination of optimal partition configurations based on available on-chip memory resources before execution. This preliminary action enables the system to pre-calculate partition boundaries and data flow paths, reducing runtime complexity and energy consumption by avoiding dynamic memory management operations
2Productivity
If depth-first processing with partitioning is implemented, then DCN performance improves by reducing off-chip memory access, but computational complexity increases due to partition management
Solution Approach 1:
The computational graph is segmented into discrete partitions that can be independently processed and cached in on-chip memory. This segmentation improves DCN performance by minimizing repeated off-chip memory accesses while managing computational complexity through localized processing within each partition
Solution Approach 2:
Multiple partitions are nested within a hierarchical memory structure where smaller on-chip buffers are nested within larger off-chip memory. This nesting allows the system to optimize performance by keeping frequently accessed partition data in faster on-chip memory while maintaining access to larger datasets in off-chip memory when needed
3Quantity of substance
If partitioning is determined based on on-chip memory availability, then memory utilization efficiency improves, but processing time increases due to partition determination overhead
Solution Approach 1:
Partition configurations are determined in advance based on available on-chip memory resources, creating a static partitioning scheme that maximizes memory utilization. This preliminary determination eliminates runtime partition calculation overhead, improving processing speed while maintaining high memory utilization efficiency through pre-optimized partition boundaries
Solution Approach 2:
The system adjusts partition parameters such as layer groupings and buffer sizes based on available on-chip memory capacity. By changing these parameters to match actual hardware resources, the system achieves optimal memory utilization without requiring complex runtime adjustments that would increase processing time
Data Source
AI summary
A method performed by a computing device includes determining a partition for depth-first processing by a multi-layer artificial neural network (ANN) of the computing device. The computing device comprising a processor, on-chip memory, and off-chip memory. The first partition determined based on an amount of on-chip memory used by the first partition, an available amount of on-chip memory, and a size of a write back to the off-chip memory. The method also includes processing, at the device via the multi-layer ANN, an input, using the depth-first processing in accordance with the partition.