Vectorized Lossless Compression for High-Rate Data Streams

Resolve Bottlenecks,
Find Innovative Solutions
Generate Solutions

Solution Overview

Problem

Current lossless compression algorithms for high-speed data streams, such as those from astroparticle physics experiments, face challenges in achieving fast compression speeds while maintaining high compression ratios, which is crucial for handling large data volumes efficiently and reducing I/O bottlenecks in data storage and processing.

Innovation Solution

A method that maps digital data values to code words with non-uniform bit lengths, using morphological wavelet transforms and adaptive bit length selection through binary search, allowing for efficient packing and decoding of data, optimized for parallel processing with AVX2 instructions.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of substance

If general-purpose lossless compression algorithms like gzip are used, then good compression ratios are achieved, but compression speed becomes insufficient

Engineering Contradiction:
Improvedata sizeVSAvoidcompression speed
Core Design Contradiction:
Loss of substanceVSProductivity

Solution Approach 1:

The patent segments the compression process into distinct stages: prediction residual calculation, run-length encoding, and variable-bit encoding. Each stage operates independently and can be optimized separately, with the residual calculation stage processing data in parallel blocks that feed into the encoding stages, thereby achieving both high compression ratios and fast compression speeds

Inventive Principle:
Principle #1Segmentation

Solution Approach 2:

The patent replaces traditional mechanical compression mechanisms with morphological wavelet transforms and adaptive prediction algorithms. These mathematical transforms efficiently capture data correlations and redundancies without requiring complex mechanical processing, enabling rapid compression of high-rate data streams while maintaining lossless compression ratios

Inventive Principle:
Principle #28Mechanics substitution (Replace mechanical system)

2Speed

If compression is applied to high-rate data streams, then I/O speed is improved, but compression algorithm complexity increases

Engineering Contradiction:
ImproveI/O speedVSAvoidalgorithm complexity
Core Design Contradiction:
SpeedVSDevice complexity

Solution Approach 1:

The patent changes key parameters of the compression algorithm to optimize for speed: using fixed prediction windows instead of adaptive ones, employing simple run-length encoding rather than complex dictionary methods, and using variable-bit encoding with predetermined bit allocations. These parameter changes reduce computational complexity while maintaining effective compression for high-rate data streams

Inventive Principle:
Principle #35Parameter changes

Solution Approach 2:

The patent performs preliminary prediction and residual calculation on data blocks before the main encoding process. By pre-processing the data to extract prediction residuals and organize them into run-length sequences, the algorithm reduces the complexity of the subsequent encoding stages, enabling faster real-time compression of high-rate streams

Inventive Principle:
Principle #10Preliminary action

Data Source

PatentUS12119844B2Method and device for fast lossless compression
Publication Date: 2024.10.15 MAX PLANCK GESELLSCHAFT ZUR FOERDERUNG DER WISSENSCHAFTEN EV
  • US12119844B2 patent drawing
  • US12119844B2 patent drawing
  • US12119844B2 patent drawing

AI summary

A computer-implemented method for compressing digital data includes obtaining a sequence of digital data values; mapping the sequence of digital data values to a sequence of code words having non-uniform bit lengths; packing the sequence of code words into a sequence of storage words having uniform bit length and corresponding to a fixed-size piece of data handled as a unit by the instruction set or the hardware of a processor; and outputting the sequence of storage words together with a first bitmask indicating the bit length of each code word, wherein in the method is implemented using special purpose vector instructions.