DNN Weight Bit Interleaving for Accurate Model Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge in B5G/6G communications lies in effectively compressing deep neural network (DNN) models for transmission while preserving model accuracy, as existing compression techniques often reduce bit precision and introduce errors due to the large and intricate nature of DNN models.
Innovation Solution
A data processing system and method that quantizes DNN model weights into different types of bits, applies interleaving and compression techniques based on bit type, and uses channel coding to generate encoded data for efficient transmission, optimizing resource utilization while maintaining prediction accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If compression techniques are applied to reduce bit precision of DNN model parameters, then transmission efficiency and storage requirements are improved, but model accuracy deteriorates due to introduced errors
Solution Approach 1:
The patent segments the 32-bit floating-point parameters into four distinct bit sets based on their compressibility and importance: first-type bits (most significant, non-compressible), second-type bits (compressible, less important), third-type bits (compressible, important), and fourth-type bits (least important). This segmentation allows different compression strategies to be applied to different bit sets, achieving both compression and accuracy preservation.
Solution Approach 2:
The patent applies different compression rates to different bit sets based on their local characteristics. The first bit set uses no compression (maintaining full precision for critical bits), while the second and third bit sets use different compression rates (e.g., 4:1 or 8:1) appropriate to their compressibility and importance. This local quality approach ensures that critical information is preserved while less critical information is compressed.
2Quantity of substance
If uniform compression rate is applied to all bit sets, then transmission resources are optimized, but important information may be lost reducing model performance
Solution Approach 1:
The patent divides the parameter bits into four segments with different importance levels and compressibility characteristics. The first bit set contains the most important non-compressible bits, while the second and third bit sets contain compressible bits of varying importance. This segmentation enables selective compression that preserves information fidelity for critical data while reducing overall transmission volume.
Solution Approach 2:
The patent changes the compression parameter (compression rate) based on the bit set type. The first bit set uses a compression rate of 1:1 (no compression), while the second and third bit sets use higher compression rates (e.g., 4:1, 8:1). This parameter change approach allows the system to optimize transmission volume while maintaining information fidelity for important bits.
3Volume of stationary object
If bit precision is reduced for compression, then storage requirements are decreased, but transmission reliability deteriorates due to error introduction
Solution Approach 1:
The patent segments parameters into bit sets with different compression characteristics. The first bit set (most significant bits) is excluded from compression or uses minimal compression, ensuring transmission reliability for critical information. The second and third bit sets use higher compression rates, reducing storage requirements while maintaining acceptable reliability through the interleaving and error correction mechanisms.
Solution Approach 2:
The patent applies block interleaving before compression and channel coding after compression to provide error protection. The interleaving distributes bits across multiple symbols, and the channel coding adds redundancy, creating a cushion against transmission errors. This beforehand cushioning allows the system to use higher compression rates while maintaining transmission reliability.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A data processing method (5) for a DNN model (110) includes: reading weights (W1-Wn) of transmission data (21); quantizing each weight into bits sequentially including first, second, third, and fourth-type bits; sequentially interleaving the first-type bit into a first bit set (26); sequentially interleaving each second-type bit into second bit sets (27A-27D) and reading a second compression rate in response to the compressible second bit sets; interleaving the third-type bit into a third bit set (28) and reading a third compression rate in response to the compressible third bit set; compressing each second bit set with the second compression rate, and compressing the third bit set with the third compression rate; sequentially coding the first bit set, each compressed second bit set, and the compressed third bit set to generate first encoded data (30) corresponding to the transmission data; transmitting the first encoded data to an external device (3).