3D Tree Coding for Neural Network Compression With Lower Buffer Use
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network compression methods, such as the Coded Representation of Neural Network (NNR), fail to optimize memory access patterns for inference operations on mobile and edge devices, leading to excessive buffer requirements and inefficient entropy coding after partitioning weight tensors.
Innovation Solution
A 3D-Tree coding method is employed to reshape and partition neural network parameters into 3D coding tree units (CTU3Ds) using quad-trees, followed by entropy encoding, which adapts block partitioning and reordering to optimize memory access and reduce buffer size.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If weight tensors are partitioned using existing NNR methods, then compression is achieved, but memory access patterns are not optimized leading to excessive buffer requirements
Solution Approach 1:
The patent segments the weight tensor into multiple blocks along different dimensions (input features, output features, kernel dimensions). This segmentation allows independent processing of each block, optimizing memory access patterns during inference while maintaining compression efficiency. The tensor is divided into manageable chunks that can be processed sequentially with reduced buffer requirements.
Solution Approach 2:
The patent introduces a new dimensionality approach by organizing weight blocks in a multi-dimensional structure that optimizes memory access patterns. By rearranging the tensor dimensions and introducing block-level indexing, the method creates more efficient access paths during inference operations, reducing the working buffer size needed while maintaining computational efficiency.
2Quantity of substance
If weight tensors are compressed using existing methods, then storage is reduced, but entropy coding efficiency is insufficient
Solution Approach 1:
The patent applies local quality optimization by performing entropy coding on individual weight blocks rather than the entire tensor. Each block can be coded with locally optimized parameters and statistics, improving overall compression efficiency. This block-level approach allows adaptive coding strategies tailored to the specific characteristics of each weight block, enhancing the compression ratio while maintaining accuracy.
3Quantity of substance
If buffer size is reduced for resource-constrained devices, then memory requirements are met, but inference operation efficiency may be impacted
Solution Approach 1:
The patent performs preliminary organization and ordering of weight blocks before inference operations. By pre-arranging the blocks in an access-optimized sequence and pre-computing necessary indexing structures, the method enables efficient inference with reduced buffer size. This preliminary preparation eliminates the need for large working buffers during actual inference, making operations feasible on resource-constrained devices.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method of three-dimensional (3D)-Tree coding for neural network model compression, is performed by at least one processor, and includes reshaping a four-dimensional (4D) parameter tensor of a neural network into a 3D parameter tensor of the neural network, the 3D parameter tensor comprising a convolution kernel size, an input feature size, and an output feature size, partitioning the 3D parameter tensor along a plane that is formed by the input feature size and the output feature size into 3D coding tree units (CTU3Ds), partitioning each of the CTU3Ds into a plurality of 3D coding units (CU3Ds) recursively until a predetermined depth, using a quad-tree, constructing a 3D tree for each of the plurality of CU3Ds, and entropy encoding each of a plurality of values of a plurality of nodes of the 3D tree