Inter-Frame Feature Map BFP Compression for Stateful Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Stateful inference in neural networks requires significant memory footprint, which is a challenge in environments with limited resources such as wearable or mobile devices, leading to potential performance restrictions.
Innovation Solution
Implement segment-wise compression and decompression of feature map segments using block floating-point (BFP) format to reduce memory footprint and energy consumption, leveraging temporal sparsity by updating only significant changes between frames.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If stateful inference is implemented in neural networks, then temporal context and sequential data processing capability are improved, but memory footprint increases significantly
Solution Approach 1:
The feature maps are divided into multiple segments that can be processed and stored separately. This segmentation allows the system to manage memory resources more efficiently by only loading and processing relevant segments, thereby reducing the overall memory footprint while maintaining stateful inference capabilities.
Solution Approach 2:
The patent employs block floating-point (BFP) format to change the parameter representation of feature map values. By using BFP instead of standard floating-point formats, the system achieves more compact storage with reduced precision requirements, significantly decreasing memory footprint while preserving sufficient accuracy for temporal context processing.
2Speed
If feature maps are stored in uncompressed format, then processing speed and accuracy are improved, but memory consumption increases
Solution Approach 1:
The patent changes the numerical representation parameters by using block floating-point format with reduced precision. This allows feature maps to be stored more compactly while maintaining adequate processing speed and accuracy by performing decompression and updating operations only on active segments rather than entire feature maps.
Solution Approach 2:
Instead of compressing and decompressing entire feature maps, the system applies partial action by only processing and updating active segments that contain relevant information. This reduces both memory consumption and processing overhead, as the system performs operations only where necessary rather than on the complete feature map.
3Quantity of substance
If entire feature maps are compressed and decompressed, then memory footprint is reduced, but processing time and energy consumption increase
Solution Approach 1:
The patent segments feature maps into multiple regions, allowing the system to compress and decompress only the active segments that contain relevant information rather than the entire feature map. This segmentation strategy significantly reduces processing time and energy consumption while maintaining reduced memory footprint benefits.
Solution Approach 2:
The system performs partial compression and decompression operations on only the active segments of feature maps rather than processing the complete feature map. This partial action approach minimizes processing time and energy expenditure while still achieving effective memory footprint reduction through selective compression of relevant data portions.
Data Source
AI summary
Examples described herein relate to stateful inference of a neural network. A plurality of feature map segments each has a first set of values stored in a compressed manner. The first sets of values at least partially represent an extrinsic state memory of the neural network after processing of a previous input frame. Operations are performed with respect to each feature map segment. The operations include decompressing and storing the first set of values. The operations further include updating at least a subset of the decompressed first set of values based on a current input frame to obtain a second set of values. The second set of values is compressed and stored. Memory resources used to store the decompressed first set of values is released. The second sets of values at least partially represent the extrinsic state memory of the neural network after processing of the current input frame.