Columnar Data Transformations for Faster Compression and Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face challenges in achieving an optimal balance between storage space and processing performance due to inefficient data compression and decompression methods, especially when handling data with varying data types and formats, leading to sub-optimal storage usage and processing speeds.
Innovation Solution
The implementation of data transformation techniques such as byte-wise differential transformation, length separation transformation, native number transformation, and native datetime-type transformation, which convert data into more compressible forms by computing differences, separating lengths, removing redundant data, and encoding datetime values, respectively, to enhance compression ratios and processing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored in compressed form, then storage space is reduced, but processing speed deteriorates due to required compression and decompression operations
Solution Approach 1:
The patent applies compression transformations to data before storage, preparing it in an optimized format ahead of time. This preliminary compression allows the data to be stored efficiently while minimizing the need for compression operations during retrieval, as the transformed data structure maintains compressibility characteristics.
Solution Approach 2:
The patent transforms data by changing its parameters through various compression techniques (byte-wise differential, length separation, native number, native datetime transformations). These parameter changes convert original data into transformed representations that occupy less space while preserving the ability to be processed efficiently when needed.
2Quantity of substance
If data transformation is applied to improve compression, then storage efficiency is improved, but computational resources increase during transformation
Solution Approach 1:
The computationally intensive transformation operations are performed in advance during data ingestion or ETL processes, rather than during query execution. This shifts the computational burden to batch processing scenarios where resources can be allocated more efficiently, leaving query operations to work with already-transformed data.
Solution Approach 2:
The patent divides data into separate columns and applies specific transformation techniques to each column type (e.g., native number transformation for numeric columns, native datetime transformation for temporal data). This segmentation allows transformations to be applied selectively and independently, reducing overall computational overhead compared to transforming entire datasets uniformly.
Data Source
AI summary
Described herein are compression and processing optimizations by using data transformation techniques. In example embodiments, a byte-wise differential transformation is applied to columnar data represented as a list of length-value pairs to determine a list of delta pairs that is subsequently compressed and stored on persistent storage. A length separation transformation is applied to separate a list of length-value pairs into a length array and a corresponding data value array, where these two arrays are subsequently compressed and stored separately on persistent storage. A native number transformation is applied to a set of number values to remove the lengths stored in the number values, where the transformed set is stored on persistent storage instead of the original set of number values. A native datetime-type transformation is applied to a set of datetime values to generate an encoding that is used to encode the set of datetime values into an encoded set that is stored on persistent storage instead of the original set.