Compressed Column Slabs for Faster Large-Scale Database Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face limitations in processing speed due to hardware constraints, data storage methods, and restricted co-processing options, which hinder efficient execution of large-scale data operations.
Innovation Solution
A database system utilizing compressed column slabs, where data is segmented into segments based on redundancy storage encoding schemes, restructured using key fields, and optimized for storage and retrieval through parallelized sub-systems, enabling efficient query processing and response.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored in traditional row-based format, then storage simplicity is maintained, but query processing speed deteriorates due to inability to efficiently retrieve specific columns
Solution Approach 1:
The patent segments data into column-based units called column slabs, where each slab contains data for a single column across multiple rows. This segmentation enables independent processing and retrieval of specific columns without reading entire rows, directly improving query processing speed while managing complexity through structured organization.
Solution Approach 2:
The patent transitions from traditional row-based storage to column-based storage, effectively changing the dimensional organization of data. This dimensional shift allows the system to optimize for columnar retrieval operations, enabling faster query processing by accessing only the required columns rather than entire rows.
2Speed
If hardware resources are increased to improve processing speed, then query execution speed improves, but system cost and complexity increase
Solution Approach 1:
The patent changes the fundamental parameter of data organization from row-based to column-based storage. This parameter change enables more efficient use of existing hardware resources by improving data locality and reducing I/O operations, thereby increasing processing speed without requiring proportional increases in hardware resources.
Solution Approach 2:
The patent replaces traditional mechanical row-based storage access patterns with a columnar storage mechanism that enables parallel processing and efficient column retrieval. This substitution fundamentally changes how data is accessed and processed, improving speed while reducing the need for additional hardware resources.
3Quantity of substance
If data is compressed to reduce storage space, then storage efficiency improves, but decompression time and processing overhead increase
Solution Approach 1:
The patent performs compression on column slabs during the data loading and storage phase, preparing data in a compressed state before queries are executed. This preliminary compression action reduces storage space requirements while ensuring that decompression is only performed when data is retrieved, minimizing the impact on query processing time.
Solution Approach 2:
The patent applies compression selectively to different column slabs based on their specific characteristics and compression suitability. This local quality approach allows the system to optimize storage efficiency for compressible columns while maintaining faster access for columns where compression would be detrimental, thereby balancing storage efficiency with processing speed.
Data Source
AI summary
A database system is operable to determine a dataset for storage that includes, for each row, a plurality of column values corresponding to a plurality of columns. A plurality of column slab data are generated from the dataset, where each of the plurality of column slab data includes column values for one of the plurality of columns. Each of a plurality of compression dictionaries are trained from a corresponding one of the plurality of column slab data. Each segment of the plurality of segments is generated based on writing a set of compressed column slabs to the each segment based on compressing each of a corresponding set of the plurality of column slab data as a set of variable-length compression frames written to the each segment based on applying a corresponding one of the plurality of compression dictionaries.