Database Data Slab Compression for Faster Distributed Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database systems face limitations in processing speed due to hardware constraints, data storage methods, and restricted co-processing options, which hinder efficient data processing and query execution.
Innovation Solution
The implementation of a parallelized database system architecture that includes a parallelized data input, query, and storage subsystem, utilizing global dictionary compression (GDC) and run length encoding to optimize data storage and retrieval, allowing for efficient data compression and decompression, and enabling parallel processing across multiple nodes and cores.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is stored in traditional uncompressed format, then data retrieval speed is maintained, but storage efficiency is poor and processing time increases
Solution Approach 1:
The patent applies preliminary action by pre-compressing data using run length encoding and null elimination before storage. This preprocessing step transforms the data into a more efficient format that reduces both storage requirements and query execution time, as the compressed structure enables faster scanning and processing during retrieval operations
Solution Approach 2:
The patent changes the parameter of data representation by transforming raw data into compressed form using run length encoding (changing consecutive repeated values into count-value pairs) and null elimination (removing null values and tracking their positions). These parameter changes reduce the volume of data to be processed and stored, directly improving productivity while reducing query time
2Speed
If more hardware resources are allocated to database systems, then processing speed improves, but system cost and complexity increase
Solution Approach 1:
The patent substitutes mechanical/hardware solutions with algorithmic/software solutions. Instead of allocating more hardware resources to improve processing speed, the invention uses software-based compression algorithms (run length encoding and null elimination) to achieve faster processing and reduced storage requirements, thereby avoiding increased system complexity and cost
3Quantity of substance
If traditional storage methods are used, then data integrity is maintained, but storage space efficiency is poor
Solution Approach 1:
The patent introduces intermediaries (compression metadata and null position tracking structures) that mediate between the compressed data format and the original data representation. These intermediaries enable efficient storage by eliminating redundancy while ensuring data integrity during retrieval, as the metadata provides the necessary information to reconstruct the original data accurately
Solution Approach 2:
The patent changes the parameter of data density by applying run length encoding to compress consecutive repeated values and null elimination to remove null entries. These transformations significantly improve storage capacity utilization while maintaining data reliability through structured compression that preserves the ability to accurately retrieve and reconstruct original data
Data Source
AI summary
A method includes determining a data set for storage that includes a plurality of uncompressed data slabs in accordance with a serialized data slab ordering. A storage data set that includes a plurality of compressed data slabs is created based on the data set in accordance with the serialized data slab ordering. Each compressed data slab of the plurality of compressed data slabs is generated from at least one corresponding uncompressed data slab of the plurality of uncompressed data slabs that includes a plurality of values based on generating compressed data for each compressed data slab based on the at least one corresponding uncompressed data slab, and generating compression information for each compressed data slab. The storage data set is stored via a plurality of computing devices.