Reference-Value Difference Encoding for Low-Latency Data Transfer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing demand for efficient data transmission in applications like high-performance computing and AI model training necessitates reducing the amount of data exchanged between devices to improve efficiency and reduce latency.
Innovation Solution
A data processing method that compresses to-be-transmitted data by determining a reference value from the data, calculating differences between data pieces and the reference value, and encoding these differences into shorter codes, resulting in compressed data with reduced length.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If data is transmitted in original format, then data transmission completeness is ensured, but data transmission volume is large and latency is high
Solution Approach 1:
The patent applies parameter changes by transforming the data representation format. Specifically, it converts original data values into difference values relative to a reference value, and then encodes these differences using a code table. This parameter transformation reduces the number of bits required to represent each data element, thereby reducing transmission volume and latency while maintaining data completeness through reversible encoding.
2Quantity of substance
If data compression is applied, then data transmission volume is reduced, but data processing complexity increases
Solution Approach 1:
The patent applies preliminary action by pre-defining a code table that maps difference values to compact codes before the actual compression process. The reference value and code table are determined in advance, allowing the compression operation to simply look up and replace values rather than performing complex real-time calculations. This pre-computation approach simplifies the runtime processing complexity while achieving effective compression.
3Productivity
If floating point numbers are compressed, then data transmission efficiency is improved, but precision requirements must be maintained
Solution Approach 1:
The patent applies copying by creating a simplified representation of floating point numbers through the difference encoding process. Instead of transmitting the full precision floating point values, it transmits compact codes that copy the essential information relative to a reference value. The receiving端 can reconstruct the original precision values by applying the inverse transformation, thus maintaining measurement precision while improving transmission efficiency.
Data Source
AI summary
The present disclosure relates to data processing methods. In one example method, a computing device determines a first reference value based on a plurality of pieces of to-be-compressed first data, calculates a difference between each piece of first data and the first reference value, determines a code corresponding to each difference, and obtains compressed data of each corresponding piece of first data based on each code. The reference value is data with a larger quantity of occurrence times in the plurality of pieces of to-be-compressed data. When the difference between the reference value and each of the plurality of pieces of to-be-compressed data is calculated, a quantity of 0s in a plurality of obtained differences is the largest. After the difference is encoded, the data corresponding to the difference can be compressed to a smaller quantity of bits.