Tensor Encoding Split Inference With Multi-Scale Feature Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing CNN architectures face challenges in efficiently distributing processing between edge devices and cloud servers due to high computational complexity and the need for effective tensor data compression, which can result in accuracy loss during feature compression.
Innovation Solution
A method and system for encoding and decoding tensors using a bottleneck encoder and decoder to reduce tensor dimensions, allowing partial processing on edge devices and completing processing in the cloud, while preserving spatial detail through separate multi-scale feature compression techniques.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If tensor data is compressed for distributed processing between edge devices and cloud servers, then bandwidth consumption and computational complexity are reduced, but accuracy loss occurs during feature compression
Solution Approach 1:
The patent applies parameter changes by modifying the tensor data representation through quantization (changing precision from float32 to int8 or lower) and dimensionality reduction (changing spatial resolution). These parameter transformations enable compression while maintaining acceptable accuracy through careful optimization of quantization bits and resolution levels.
Solution Approach 2:
The patent segments the CNN processing into two parts: feature extraction performed on edge devices and classification performed on cloud servers. This segmentation allows local preprocessing with compressed data transmission, reducing bandwidth consumption while preserving accuracy for the critical classification task.
2Speed
If tensor dimensions are reduced for efficient transmission, then processing speed and bandwidth utilization improve, but information loss occurs during compression
Solution Approach 1:
The patent extracts only the essential features needed for accurate classification while discarding redundant information. The bottleneck encoder extracts salient features from high-dimensional tensors, creating a compressed representation that retains critical information for downstream classification tasks.
Solution Approach 2:
The patent transforms the tensor representation by changing dimensions through spatial downsampling and channel reduction. This dimensionality change compresses the data while preserving the most important feature relationships needed for accurate classification.
3Quantity of substance
If feature compression is applied to reduce data size, then memory consumption and transmission costs decrease, but manufacturing precision of the compressed data structure deteriorates
Solution Approach 1:
The patent systematically changes data precision parameters from high-precision float32 to lower-precision int8 or int4 formats. This parameter transformation reduces data size while maintaining sufficient precision for classification accuracy through optimized quantization schemes.
Solution Approach 2:
The patent uses lower-precision integer representations that are computationally cheaper and require less memory bandwidth. These compressed representations are processed efficiently on resource-constrained edge devices, providing a cost-effective solution for distributed inference.
Data Source
AI summary
A system and method of encoding a tensor related to image data into a bitstream. The method comprises acquiring a first tensor for the image data, the first tensor derived using a portion of a neural network, the neural network including at least a plurality of layers of a first type, each layer of the first type having at least a convolutional module and a batch-normalization module, wherein the first tensor corresponds to a tensor for which the convolutional module of one of the plurality of layers of the first type has been performed but for which the batch-normalization module of the one of the plurality of layers of the first type has not been performed. The method further comprises performing predetermined processing on the first tensor to derive a second tensor, and encoding the second tensor into the bitstream.