Neural Network Compression for Layer-by-Layer Edge Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks (DNNs) are large in size and require significant memory and processing capabilities for execution, making it challenging for limited resource devices such as phones, tablets, and IoT devices to store and execute them.
Innovation Solution
The method involves compressing a machine learning network model into a smaller form while maintaining its multiple layer structure, and generating a model file that includes the compressed model and decoding information for layer-by-layer decompression and execution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a deep neural network model is stored in full size, then the model maintains complete processing capability and accuracy, but the memory requirements and device complexity become too large for limited resource devices
Solution Approach 1:
The patent divides the deep neural network model into multiple layers and processes them sequentially. Each layer is decompressed and executed one at a time rather than loading the entire model into memory simultaneously. This segmentation allows the system to process large models on devices with limited memory by only holding one layer in memory at a time.
Solution Approach 2:
The patent implements a nested structure where compressed model data is contained within a container format that includes headers and metadata. The compressed layers are nested within this container, and the system navigates through this nested structure to access and decompress individual layers as needed, enabling efficient memory management.
2Productivity
If the entire neural network model is decompressed at once, then all layers are available for processing, but the memory consumption and processing time increase significantly
Solution Approach 1:
The patent performs preliminary compression of the model layers during the model preparation phase, storing them in a compressed format within the container. This preliminary action reduces the size of data that needs to be handled during inference, allowing for faster loading and execution on resource-constrained devices without sacrificing model accuracy.
Solution Approach 2:
The system implements periodic decompression and execution of model layers in a sequential manner. Instead of decompressing all layers at once, the system decompresses one layer, executes it, then moves to the next layer in a periodic cycle. This approach balances memory usage and processing efficiency by keeping only the currently needed layer in memory.
3Quantity of substance
If the model is compressed to reduce size, then memory requirements decrease, but the device must have capability to decompress and execute the model layer-by-layer
Solution Approach 1:
The patent introduces a container format with standardized headers and metadata as an intermediary between the compressed model data and the execution engine. This intermediary structure provides a uniform interface that simplifies the decompression and execution process, making it easier for devices to handle compressed models without requiring complex custom processing logic for each model format.
Data Source
AI summary
A first aspect relates to a computer-implemented method for performing model compression. The method includes compressing a machine learning (ML) network model comprising a multiple layer structure to produce a compressed ML network model. The compressed ML network model maintains the multiple layer structure of the ML network model. The method generates a model file for the compressed ML network model. The model file includes the compressed ML network model and decoding information for enabling the ML network model to be decompressed and executed layer-by-layer.