Lossless Data Compression with Wavelet Encoding for High I/O Speed
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing lossless compression algorithms struggle to achieve high compression speeds necessary for handling high data rates in scientific experiments, such as astroparticle physics, where data needs to be processed in real-time.
Innovation Solution
The method employs a computer-implemented approach using morphological wavelet transforms and bit range encoding to compress consecutive unsigned 16-bit integers, allowing for high parallelization and efficient use of AVX2 instructions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If general-purpose lossless compression algorithms like gzip are used, then good compression ratios are achieved, but compression speed is insufficient
Solution Approach 1:
The patent divides the compression task into processing blocks of consecutive unsigned integers. Each block is handled independently through morphological wavelet transform and bit range encoding, enabling parallel processing and significantly improving compression speed while maintaining effective compression ratios for correlated scientific data
Solution Approach 2:
The patent transforms the data representation by applying morphological wavelet transforms to convert consecutive integers into a domain where compression is more effective. It also dynamically determines bit ranges for encoding residuals, adapting the encoding parameters to the actual data characteristics to achieve both speed and compression efficiency
2Productivity
If compression is applied to high data rate streams, then I/O speed is improved, but compression speed must be sufficiently high to handle the input data rate
Solution Approach 1:
By segmenting the data stream into blocks and processing them independently with morphological wavelet transforms, the patent enables high-speed parallel compression that can keep up with high data rate inputs while reducing the compressed data volume to improve I/O speed
Solution Approach 2:
The patent replaces traditional mechanical compression algorithms with a computational approach using morphological wavelet transforms and bit range encoding, leveraging modern CPU capabilities to achieve compression speeds sufficient for high data rate scientific data streams
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A computer-implemented method for compressing digital data, comprises the steps: - obtaining a sequence of digital data values; - mapping the sequence of digital data values to a sequence of code words having non-uniform bit lengths; - packing the sequence of code words into a sequence of storage words having uniform bit length; - outputting the sequence of storage words. According to the invention, the method further comprises the step of outputting information indicating the bit length of each code word.