Atmospheric Data Compression Using Dictionary and Clustered Encoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data compression techniques, particularly for GRIB2 format atmospheric data, are inadequate in achieving significant reduction in storage size, leading to storage resource strain and inefficiencies in data handling and transmission.
Innovation Solution
A system and method that converts GRIB2 format data into floating-point format, clusters data segments, and uses dictionary encoding followed by Run Length Encoding and 7z compression to generate a complete compressed dataset, allowing for efficient and cost-effective storage while maintaining near-lossless data integrity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If lossless compression is used to preserve 100% of information, then data accuracy is maintained, but compression ratio is very low and storage space is not significantly reduced
Solution Approach 1:
The patent segments the data compression process into multiple stages: first converting GRIB2 data to floating-point format, then applying dictionary encoding to identify and encode repeated patterns, followed by run-length encoding for consecutive identical values, and finally 7z compression. This multi-stage segmentation allows each method to optimize for specific data characteristics, achieving both high compression ratios and near-lossless accuracy.
Solution Approach 2:
The patent changes the data representation parameters by converting from GRIB2 binary format to floating-point format, which enables better compression. It also adjusts compression parameters by implementing custom dictionary encodings tailored to atmospheric data patterns, and tuning the 7z compression settings to achieve optimal balance between compression ratio and data fidelity.
2Quantity of substance
If GRIB2 format is used for data storage and transmission, then data compression is achieved, but further compression is not possible and data interpretation requires external tables
Solution Approach 1:
Instead of trying to compress GRIB2 data directly (which doesn't work), the patent inverts the approach by first converting GRIB2 data to floating-point format, then applying compression. This reversal enables compression while maintaining data interpretability, as floating-point format is human-readable and doesn't require external tables for interpretation.
Solution Approach 2:
The patent introduces floating-point format as an intermediary between GRIB2 binary format and the final compressed format. This intermediary representation maintains data interpretability while enabling effective compression through subsequent encoding stages, solving both the compression and interpretability problems.
3Measurement precision
If high-resolution numerical models are used to improve weather and climate simulation accuracy, then model output quality is enhanced, but data size grows enormously causing storage and transmission strain
Solution Approach 1:
The patent changes the data format parameters from GRIB2 binary to floating-point, which reduces redundancy and enables better compression. The multi-stage compression process (dictionary encoding + run-length encoding + 7z) further reduces data volume while preserving the high-resolution information needed for accurate weather and climate simulations.
Solution Approach 2:
The patent segments the compression process into multiple specialized stages, each targeting specific patterns in the high-resolution model data. Dictionary encoding captures repeated atmospheric patterns, run-length encoding handles consecutive identical values, and 7z provides final compression, collectively reducing the enormous data volume while maintaining simulation accuracy.
Data Source
AI summary
The present disclosure relates to a system and method for compressing a dataset. The dataset can be divided in to a plurality of groups. Each group can be converted independently into corresponding text file using dictionary coding technique. The corresponding text files can be compressed independently into corresponding compressed files. Finally, all the corresponding compressed files can be combined together to generate a complete compressed data.