Vectorized Lossless Compression for High-Rate Data Streams
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current lossless compression algorithms for high-speed data streams, such as those from astroparticle physics experiments, face challenges in achieving fast compression speeds while maintaining high compression ratios, which is crucial for handling large data volumes efficiently and reducing I/O bottlenecks in data storage and processing.
Innovation Solution
A method that maps digital data values to code words with non-uniform bit lengths, using morphological wavelet transforms and adaptive bit length selection through binary search, allowing for efficient packing and decoding of data, optimized for parallel processing with AVX2 instructions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If general-purpose lossless compression algorithms like gzip are used, then good compression ratios are achieved, but compression speed becomes insufficient
Solution Approach 1:
The patent segments the compression process into distinct stages: prediction residual calculation, run-length encoding, and variable-bit encoding. Each stage operates independently and can be optimized separately, with the residual calculation stage processing data in parallel blocks that feed into the encoding stages, thereby achieving both high compression ratios and fast compression speeds
Solution Approach 2:
The patent replaces traditional mechanical compression mechanisms with morphological wavelet transforms and adaptive prediction algorithms. These mathematical transforms efficiently capture data correlations and redundancies without requiring complex mechanical processing, enabling rapid compression of high-rate data streams while maintaining lossless compression ratios
2Speed
If compression is applied to high-rate data streams, then I/O speed is improved, but compression algorithm complexity increases
Solution Approach 1:
The patent changes key parameters of the compression algorithm to optimize for speed: using fixed prediction windows instead of adaptive ones, employing simple run-length encoding rather than complex dictionary methods, and using variable-bit encoding with predetermined bit allocations. These parameter changes reduce computational complexity while maintaining effective compression for high-rate data streams
Solution Approach 2:
The patent performs preliminary prediction and residual calculation on data blocks before the main encoding process. By pre-processing the data to extract prediction residuals and organize them into run-length sequences, the algorithm reduces the complexity of the subsequent encoding stages, enabling faster real-time compression of high-rate streams
Data Source
AI summary
A computer-implemented method for compressing digital data includes obtaining a sequence of digital data values; mapping the sequence of digital data values to a sequence of code words having non-uniform bit lengths; packing the sequence of code words into a sequence of storage words having uniform bit length and corresponding to a fixed-size piece of data handled as a unit by the instruction set or the hardware of a processor; and outputting the sequence of storage words together with a first bitmask indicating the bit length of each code word, wherein in the method is implemented using special purpose vector instructions.


