Neural Network Weight Compression Using Adaptive Coding Contexts

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing neural networks require significant computational resources due to their large parameter sizes, and existing relevance estimation methods do not effectively balance compression and prediction accuracy during quantization.

Innovation Solution

Implement context-dependent arithmetic coding for encoding and decoding weight parameters of neural networks, selecting contexts based on previously encoded/decoded parameters to optimize compression.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of substance

If quantization is applied to compress neural network parameters, then compression ratio is improved, but prediction accuracy deteriorates

Engineering Contradiction:

Improveparameter sizeVSAvoidprediction accuracy

Core Design Contradiction:

Loss of substanceVSManufacturing precision

Solution Approach 1:

The patent applies different quantization bit depths to different layers or groups of layers based on their importance and compression requirements. Critical layers maintain higher precision while less critical layers use lower precision, achieving local optimization of the quality-compression tradeoff.

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The patent introduces dynamic quantization where the quantization parameters (such as bit depth and scaling factors) are adjusted based on the actual data distribution and importance of different parameters. This allows the system to adaptively balance compression and accuracy rather than using fixed quantization settings.

Inventive Principle:

Principle #15Dynamics

2Loss of substance

If context-dependent arithmetic coding is used, then compression efficiency is improved, but device complexity increases

Engineering Contradiction:

Improvebitstream sizeVSAvoidencoding complexity

Core Design Contradiction:

Loss of substanceVSDevice complexity

Solution Approach 1:

The patent pre-calculates and stores context models and probability distributions during an offline training phase. These pre-computed contexts are then reused during encoding, significantly reducing the real-time computational complexity while maintaining high compression efficiency.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent applies context-dependent arithmetic coding selectively to different parameter groups or layers, using full context modeling only where it provides the most benefit. This localized application reduces overall system complexity while maintaining compression efficiency for critical components.

Inventive Principle:

Principle #3Local quality

Data Source

PatentUS20250278600A1Methods and apparatuses for compressing parameters of neural networks

Publication Date: 2025.09.04 FRAUNHOFER GESELLSCHAFT ZUR FORDERUNG DER ANGEWANDTEN FORSCHUNG EV

AI summary

An encoder for encoding weight parameters of a neural network is configured to obtain a plurality of weight parameters of the neural network, to encode the weight parameters of the neural network using a context-dependent arithmetic coding, to select a context for an encoding of a weight parameter, or for an encoding of a syntax element of a number representation of the weight parameter, in dependence on one or more previously encoded weight parameters and/or in dependence on one or more previously encoded syntax elements of a number representation of one or more weight parameters, and to encode the weight parameter, or a syntax element of the weight parameter, using the selected context. Corresponding decoder, quantizer, methods and computer programs are also described.