Sparse Ordered Series Compression Using Predictive Code Words
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data compression methods are ineffective for large datasets of sparse multidimensional ordered series data, particularly spectroscopic data, as they fail to exploit the data's structural characteristics, leading to suboptimal compression ratios and increased data size.
Innovation Solution
The method involves predicting sparse multidimensional ordered series data using a predictor that calculates variables as functions of the index, assigning amplitude and hop code words based on value and frequency, and generating a compressed output with a decoder legend and code word dictionaries, specifically tailored for double precision floating point data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If conventional compression algorithms (GZIP, LZW, BZIP2) are applied to sparse multidimensional ordered series data, then the compression process is simple and general-purpose, but the compression ratio is poor and data size remains large
Solution Approach 1:
The patent segments the sparse multidimensional ordered series data into structured components: indexed data sets with explicit index (n), first variable (xn), and second variable (yn). This segmentation allows the compression algorithm to exploit the inherent structure and sparsity patterns in each component separately, achieving significantly better compression ratios compared to treating the data as an unstructured blob.
Solution Approach 2:
The patent transforms the data representation by introducing code words for the second variable (yn) based on frequency analysis. Frequently occurring values are encoded with shorter code words, while less frequent values use longer code words. This parameter change in encoding strategy directly addresses the sparsity characteristic, reducing the overall data size while maintaining exact representation.
2Quantity of substance
If lossy compression is applied to reduce data size, then compression ratio improves, but data accuracy and reliability deteriorate
Solution Approach 1:
The patent creates a compressed representation that is mathematically equivalent to the original data through reversible transformations. The indexed data sets with code words serve as a compact copy that can be exactly decompressed back to the original sparse multidimensional ordered series data, ensuring no loss of accuracy while achieving significant size reduction.
3Quantity of substance
If specialized compression methods exploiting data structure are used, then compression ratio improves significantly, but device complexity and implementation difficulty increase
Solution Approach 1:
The patent performs preliminary organization of the sparse multidimensional ordered series data into indexed data sets before compression. By pre-structuring the data with explicit indices and variable assignments, the subsequent compression step becomes more straightforward and efficient. This preliminary action reduces the complexity of the main compression algorithm while achieving better compression ratios.
Solution Approach 2:
The patent introduces code words as an intermediary representation between the original data and the compressed form. These code words serve as a mediator that captures the essential information in a compact format, simplifying the compression process while maintaining the ability to exactly reconstruct the original data during decompression.
Data Source
AI summary
Disclosed are methods and systems for significantly compressing sparse multidimensional ordered series data comprised of indexed data sets, wherein each data set comprises an index, a first variable and a second variable. The methods and systems are particularly suited for compression of data recorded in double precision floating point format.