Partial Inference Framework for Edge DNN Memory Constraints
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Constrained edge devices with limited memory capabilities struggle to implement deep neural networks for tasks like Acoustic Scene Classification due to high memory requirements for neural network parameters.
Innovation Solution
A method and processing device that perform partial inference on input data using a neural network with at least two layers, where the parameters of each layer are stored and processed sequentially, allowing for memory-efficient processing by discarding each layer's parameters after use.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If deep neural networks with multiple layers are implemented on edge devices, then feature extraction capability and processing sophistication are improved, but memory requirements exceed the limited capabilities of constrained devices
Solution Approach 1:
The patent divides the deep neural network into multiple layers that are processed sequentially rather than loading all layers simultaneously. Each layer's parameters are loaded, processed, and then discarded before moving to the next layer, effectively segmenting the memory requirements into manageable chunks that fit within constrained edge device memory.
Solution Approach 2:
The patent performs preliminary computation by pre-calculating and storing intermediate results that can be reused across multiple layers. This allows subsequent layers to operate on pre-processed data rather than raw input, reducing the computational burden and memory requirements for each processing step.
2Productivity
If dedicated hardware like GPUs or TPUs is used for neural network processing, then processing speed and efficiency are improved, but device complexity and cost increase
Solution Approach 1:
The patent enables standard CPUs to efficiently execute neural network computations by implementing optimized algorithms and data structures that maximize the use of general-purpose processor capabilities. This self-service approach allows conventional devices to perform neural network inference without requiring specialized hardware acceleration.
Solution Approach 2:
The patent implements partial inference by processing only the necessary portion of the neural network on the edge device rather than the entire network. This allows the device to achieve sufficient processing performance for specific tasks without the overhead of implementing complete deep neural networks, reducing the need for dedicated hardware.
3Ease of operation
If all neural network parameters are loaded into memory simultaneously, then complete network processing is enabled, but memory consumption exceeds available capacity on constrained devices
Solution Approach 1:
The patent implements a streaming approach where neural network parameters for each layer are loaded into memory, processed through the computation, and then discarded to make room for the next layer's parameters. This循环利用 of memory space allows the system to handle deep neural networks with arbitrary numbers of layers using only enough memory to hold a single layer's parameters plus intermediate results.
Data Source
AI summary
The present disclosure relates to a method for performing inference on input data using a neural network and a processing device employing the aforementioned method. The method comprises the steps of obtaining and storing input data, obtaining parameter data indicating the parameters of the first layer and storing the parameter data in a parameter data storage location and processing the input data using the first layer parameter data, to form first layer output data. The method further comprises storing the first layer output data, obtaining parameter data of the second layer and storing the second layer parameter data by replacing the first layer parameter data with the second layer parameter data, processing the first layer output data using the stored second layer parameter data to form second layer output data; and storing the second layer output data.