Sparse Multidimensional Data Compression Using Correlation Subtraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing compression methods are ineffective in managing large multi-dimensional data sets, such as spectroscopic data, resulting in inefficient storage, transfer, and processing due to limited compression ratios and potential file size expansion, especially when dealing with data containing double precision floating point numbers and high-dynamic range.
Innovation Solution
A method for compressing sparse multidimensional ordered series data by dividing the data into local regions, calculating correlations between current and previous data sets, and predicting and subtracting correlated portions, followed by encoding adjusted data with an optimum scale factor.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional compression methods are applied to multi-dimensional spectroscopic data, then file size is reduced slightly, but the data fidelity is significantly degraded and processing time increases
Solution Approach 1:
The patent segments multi-dimensional spectroscopic data into multiple two-dimensional projection matrices by decomposing the N-dimensional data structure. This segmentation allows each projection to be compressed independently using conventional methods while preserving the ability to reconstruct the original high-fidelity data through inverse transformation, thus achieving both file size reduction and data fidelity preservation
Solution Approach 2:
The patent transforms N-dimensional spectroscopic data into multiple 2-dimensional projection matrices, effectively changing the dimensional representation. This dimensionality transformation enables the use of efficient 2D compression algorithms while maintaining the capacity to recover the original multi-dimensional structure with high fidelity through mathematical reconstruction
2Quantity of substance
If conventional compression methods are applied to multi-dimensional spectroscopic data, then file size is reduced slightly, but storage and transfer costs remain high due to inefficiency
Solution Approach 1:
By segmenting N-dimensional data into multiple 2D projection matrices, the patent enables parallel processing and more efficient compression of each segment. This segmentation strategy significantly improves storage and transfer efficiency compared to applying conventional compression directly to the entire multi-dimensional dataset, achieving up to 100-fold or greater compression ratios
Solution Approach 2:
The transformation to 2D projection matrices enables the use of highly optimized 2D compression algorithms and data structures, dramatically improving storage and transfer efficiency. The dimensional change allows for more compact representation and faster processing compared to traditional approaches for multi-dimensional spectroscopic data
3Quantity of substance
If compression is applied to reduce file size, then storage costs decrease, but processing time increases due to compression and decompression overhead
Solution Approach 1:
The patent segments the compression and decompression process into independent operations on 2D projection matrices. This segmentation allows for faster processing compared to compressing/decompressing the entire N-dimensional dataset as a single unit, reducing the computational overhead while achieving significant file size reduction
Solution Approach 2:
By working with 2D projection matrices instead of N-dimensional data, the patent enables the use of highly optimized 2D Fast Fourier Transform and other efficient algorithms. This dimensional change reduces processing time for both compression and decompression operations while maintaining compression effectiveness
Data Source
AI summary
Described herein are computer-implemented methods for compressing sparse multidimensional ordered series data. In particular, these methods and apparatuses for performing them (including software) may be particularly well suited to efficiently compressing spectrographic data.