Compressed Data Slabs With Metadata for Faster Database Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database systems face limitations in processing speed due to hardware constraints, data storage methods, and restricted co-processing options, which affect the efficiency of data storage and retrieval.
Innovation Solution
The implementation of a parallelized database system with sub-systems for data input, storage, query processing, and response, utilizing global dictionary compression (GDC) and run length encoding techniques to compress data, thereby optimizing storage and retrieval processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored in uncompressed format, then data retrieval speed is improved, but storage space consumption increases
Solution Approach 1:
The patent segments data into fixed-size data slabs that are compressed and stored independently in database blocks. Each data slab maintains its own compression metadata, allowing selective decompression and retrieval of specific segments without processing entire datasets, thus balancing storage efficiency with retrieval speed.
Solution Approach 2:
The patent applies compression algorithms that transform data from uncompressed to compressed format, changing the physical representation of data while maintaining logical accessibility. Compression metadata is generated to track the transformation parameters, enabling efficient decompression only when needed for queries.
2Productivity
If parallelized processing is implemented, then processing speed is improved, but system complexity increases
Solution Approach 1:
The patent divides data processing into parallel operations on separate data slabs within database blocks. Multiple compression and decompression operations can execute simultaneously on different blocks, improving throughput while maintaining manageable complexity through standardized slab structures and metadata management.
Solution Approach 2:
The patent introduces compression metadata as an intermediary layer between compressed data slabs and query processing operations. This metadata structure manages the complexity of parallel operations by providing a standardized interface for tracking compressed data locations, formats, and retrieval requirements.
3Quantity of substance
If compression algorithms are applied, then storage efficiency is improved, but processing overhead increases
Solution Approach 1:
The patent applies compression selectively at the data slab level rather than processing entire datasets uniformly. Not all data requires the same compression level - the system processes data in fixed-size slabs that can be independently compressed and stored, reducing overall processing overhead while maintaining storage efficiency.
Solution Approach 2:
The patent performs compression of data slabs during data loading and storage operations, before queries are executed. This preliminary compression reduces the size of stored data, and compression metadata is generated in advance to facilitate efficient retrieval without repeated compression processing during query operations.
Data Source
AI summary
A computing entity is operable to a data set for storage that includes a plurality of uncompressed data slabs. A storage data set that includes a plurality of compressed data slabs is created based on the data set, and the storage data set is stored via a plurality of computing devices. Each compressed data slab of the plurality of compressed data slabs is generated from at least one corresponding uncompressed data slab of the plurality of uncompressed data slabs, and each compressed data slab is generated to include compressed data and compression information. The compressed data is generated based on determining non-null ones of the plurality of values of the at least one corresponding uncompressed data slab. The compression information is generated based on positions of the non-null ones of the plurality of values in accordance with a serialized value ordering.