Sparse Ordered Series Compression for Floating-Point Spectral Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data compression methods are ineffective for large datasets of sparse multidimensional ordered series data, particularly spectroscopic data, as they fail to exploit the data's structural characteristics, leading to suboptimal compression ratios and increased data size.
Innovation Solution
The method involves defining a predictor to model the data, assigning amplitude and hop code words based on value and frequency, and generating a compressed output with a decoder legend and code word dictionaries, specifically tailored for sparse multidimensional ordered series data like mass spectrometry data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional compression algorithms (GZIP, LZW, BZIP2, etc.) are applied to sparse multidimensional ordered series data, then the data can be compressed with general-purpose methods, but the compression ratio is suboptimal and data size remains large
Solution Approach 1:
The patent transforms the data representation by changing parameters: converting absolute values to relative differences (delta encoding), transforming floating-point numbers to integers, and reorganizing multidimensional data into linear sequences. These parameter changes enable conventional compressors to achieve much better compression ratios on spectroscopic data.
Solution Approach 2:
The patent segments the large spectroscopic dataset into smaller chunks or blocks, applying compression transformations to each segment. This allows the compression algorithm to exploit local patterns and correlations within each segment while managing memory requirements efficiently.
2Reliability
If lossless compression is applied to preserve all data information, then no information is lost during compression, but the compression ratio is far lower than lossy compression
Solution Approach 1:
The patent applies parameter changes that are reversible, allowing lossless compression. By transforming data to a different representation (delta encoding, integer conversion) and storing transformation metadata, the original data can be perfectly reconstructed, achieving both high compression ratio and full data fidelity.
3Quantity of substance
If lossy compression is applied to achieve higher compression ratios, then data size is reduced more effectively, but the accuracy of the restored data cannot be guaranteed
Solution Approach 1:
The patent implements reversible parameter changes that maintain data accuracy while achieving high compression. The transformation from floating-point to integer representation, combined with delta encoding, allows lossless compression that preserves measurement precision without sacrificing compression effectiveness.
4Ease of manufacture
If general-purpose compression algorithms are used without data-specific optimizations, then the algorithms are simple to implement, but they fail to exploit the structural characteristics of spectroscopic data
Solution Approach 1:
The patent performs preliminary data transformations (delta encoding, integer conversion, flattening) before applying conventional compression algorithms. These preliminary actions prepare the data in a format that exploits its structural characteristics, enabling simple compressors to achieve superior compression ratios without complex algorithm modifications.
Data Source
AI summary
Disclosed are methods and systems for significantly compressing sparse multidimensional ordered series data comprised of indexed data sets, wherein each data set comprises an index, a first variable and a second variable. The methods and systems are particularly suited for compression of data recorded in double precision floating point format.