Compressed Column Slabs for Parallel Database Query Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face limitations in processing speed due to hardware constraints, data storage methods, and restricted co-processing options, which hinder efficient execution of large-scale data operations.
Innovation Solution
The implementation of a database system that utilizes parallelized data input, storage, and query processing systems, including segmenting data into compressed column slabs for efficient storage and query execution, utilizing a segment generator, column slab compression module, and query execution module to optimize data processing across multiple computing devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored in traditional formats, then storage capacity is adequate, but processing speed is limited due to hardware constraints
Solution Approach 1:
The patent segments data into column slabs that can be independently processed and stored. Each column slab represents a discrete unit of data that can be processed in parallel, eliminating the bottleneck of sequential processing and improving overall processing speed without requiring proportional increases in hardware complexity.
Solution Approach 2:
The patent changes the fundamental parameter of data organization from row-based to column-based storage. This parameter change enables more efficient compression ratios and allows query processing to operate on compressed data directly, significantly improving processing speed while reducing the computational burden on hardware.
2Quantity of substance
If data is compressed into column slabs, then storage efficiency improves, but data structure complexity increases
Solution Approach 1:
By segmenting data into standardized column slab units with consistent structures, the patent achieves high compression efficiency while maintaining manageable complexity. Each slab is an independent, uniformly structured unit that can be processed separately, preventing complexity from compounding across the entire dataset.
Solution Approach 2:
The patent introduces column slabs as an intermediary data structure between raw data and storage. This intermediary layer provides a standardized format that simplifies compression operations and enables efficient query processing, acting as a buffer that manages the complexity transformation from raw data to compressed storage.
3Productivity
If parallelized processing is implemented, then query execution speed improves, but system complexity increases
Solution Approach 1:
The patent segments both data and processing operations into parallelizable units. Column slabs can be processed independently in parallel, allowing query execution to distribute work across multiple processing cores or nodes. This segmentation enables linear scaling of query execution speed with the number of processing units without proportionally increasing system complexity.
Solution Approach 2:
The patent combines multiple column slabs into organized collections that can be processed as unified structures. This merging approach allows parallel processing of individual slabs while maintaining the ability to aggregate results efficiently, achieving high productivity without requiring complex coordination mechanisms between parallel processing elements.
Data Source
AI summary
A store and compute sub-system of a database system includes a plurality of computing device clusters, wherein a first computing device cluster includes a first set of computing devices that is operably coupled to receive a query regarding a plurality of records of a dataset. The query includes a filtering parameter, wherein the plurality of records is stored as a multitude of data objects within memory of the first set of computing devices and configuration data includes data regarding mapping of sets of records to data objects. Based on the configuration data and the filtering parameter, the system identifies the data objects that include at least one record that satisfies the filtering parameter data to produce identified data objects. From the identified data objects, the system identifies records that satisfy the filtering parameter and executes at least a portion of the query on the identified records to produce a first query response.