Quantization Neural Network for Federated Learning

Solution Overview

Problem

Existing methods for quantizing large numbers of values in distributed and federated learning are either inaccurate due to high statistical variance or too slow for real-time applications, necessitating a method that can efficiently and accurately reduce data transmission by determining optimal quantization levels.

Innovation Solution

Training a neural network to determine optimal quantization levels for approximating large vectors, where a smaller input layer is used to sample coordinates and output levels that can be applied to the entire vector, allowing for real-time use in distributed and federated learning environments.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If complex algorithms are used to determine optimal quantization levels, then accuracy of approximation is improved, but processing speed deteriorates making it too slow for real-time applications

Engineering Contradiction:

Improveaccuracy of coordinate approximationVSAvoidprocessing speed

Core Design Contradiction:

Measurement precisionVSProductivity

Solution Approach 1:

The patent pre-computes and stores optimal quantization levels in a lookup table during an offline training phase. During real-time distributed learning, the system only performs fast hash-based indexing and table lookups rather than complex optimization computations, thus achieving both high accuracy and real-time performance

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent divides the large gradient vector into multiple blocks or segments, and determines quantization levels for each block independently using local coordinate statistics. This segmentation reduces the computational complexity from processing all coordinates globally to processing smaller blocks locally, enabling real-time processing while maintaining accuracy

Inventive Principle:

Principle #1Segmentation

2Quantity of substance

If quantization is applied to reduce transmitted bits, then communication bandwidth requirement is reduced, but quantization accuracy deteriorates due to high statistical variance

Engineering Contradiction:

Improvenumber of transmitted bitsVSAvoidaccuracy of coordinate approximation

Core Design Contradiction:

Quantity of substanceVSMeasurement precision

Solution Approach 1:

The patent changes the parameter selection criterion from fixed or random quantization levels to levels determined by the actual statistical distribution of gradient coordinates. By using the median and quartile values from the coordinate distribution as quantization levels, the system adapts to the data characteristics, minimizing quantization error while maintaining low bit representation

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The patent replaces complex mechanical-style optimization algorithms with a statistical approach based on hash functions and lookup tables. Instead of iterative optimization, the system uses hash-based indexing to quickly retrieve pre-computed optimal levels, substituting computational mechanics with statistical probability methods that are both accurate and efficient

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

3Productivity

If simple quantization methods are used, then processing speed is improved for real-time use, but accuracy deteriorates with high statistical variance

Engineering Contradiction:

Improveprocessing speedVSAvoidaccuracy of coordinate approximation

Core Design Contradiction:

ProductivityVSMeasurement precision

Solution Approach 1:

The patent pre-computes and stores optimal quantization levels in a lookup table during an offline training phase. During real-time distributed learning, the system only performs fast hash-based indexing and table lookups rather than complex optimization computations, thus achieving both high accuracy and real-time performance

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent replaces complex mechanical-style optimization algorithms with a statistical approach based on hash functions and lookup tables. Instead of iterative optimization, the system uses hash-based indexing to quickly retrieve pre-computed optimal levels, substituting computational mechanics with statistical probability methods that are both accurate and efficient

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Data Source

PatentUS20240256893A1Learning quantization levels

Publication Date: 2024.08.01 VMWARE INC

AI summary

A method of reducing data transmission between neural networks in a distributed or federated learning environment, includes the steps of: training a quantization neural network by using a plurality of training vectors each having a dimension k, wherein the quantization neural network is configured to, based on said training, output quantization levels for approximating input vectors having the dimension k; after training the quantization neural network, randomly sampling coordinates of a vector having a dimension d, to provide a first set of k coordinates, wherein d is greater than k; inputting the first set of k coordinates to the quantization neural network to determine first quantization levels for approximating the first set of k coordinates; quantizing the vector having the dimension d based on the determined first quantization levels; and using the quantized vector in the distributed or federated learning environment.