Parallel Database Data Slab Compression for Faster Query Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face limitations in processing speed due to hardware constraints, data storage methods, and restricted co-processing options, leading to inefficiencies in data handling and query execution.
Innovation Solution
A parallelized database system architecture that utilizes a network of computing devices with multiple nodes and processing core resources, enabling lock-free and parallel execution of administrative and configuration operations, along with data partitioning and compression techniques like global dictionary compression (GDC) to optimize query processing and storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored in a traditional database system with hardware constraints, then data storage capacity is maintained, but processing speed and query execution efficiency deteriorate
Solution Approach 1:
The patent divides the database system into multiple independent nodes, each capable of autonomous operation. Data is partitioned across these nodes, allowing parallel processing of queries and administrative operations. This segmentation enables the system to overcome hardware constraints of individual devices by distributing the computational load across multiple devices, thereby improving processing speed without requiring a single complex centralized system.
Solution Approach 2:
The patent transitions from a single-dimensional centralized database architecture to a multi-dimensional distributed architecture. By adding the dimension of spatial distribution across multiple nodes and introducing parallel processing dimensions, the system achieves improved processing speed. This dimensional transformation allows administrative and configuration operations to execute in parallel without interfering with each other, resolving the speed-complexity contradiction.
2Loss of time
If parallel processing is implemented in a database system, then query execution time is reduced, but system complexity and coordination overhead increase
Solution Approach 1:
The patent performs preliminary actions by pre-defining data partitioning schemes, node roles, and communication protocols before parallel query execution begins. Configuration operations are prepared and distributed to appropriate nodes in advance. This preliminary setup eliminates the need for complex real-time coordination during query execution, as each node operates autonomously based on pre-established rules, thereby reducing query execution time without proportionally increasing system complexity.
Solution Approach 2:
Each node in the distributed database system operates autonomously, making local decisions about data storage, query processing, and configuration management without requiring constant centralized coordination. Nodes self-organize and self-manage their operations, reducing the coordination overhead that would otherwise accompany parallel processing. This self-service approach enables significant reductions in query execution time while keeping system coordination complexity manageable.
3Quantity of substance
If data partitioning and compression techniques are applied, then storage efficiency is improved, but data access and retrieval complexity increase
Solution Approach 1:
The patent implements feedback mechanisms that track data location, compression status, and access patterns across the distributed nodes. This feedback information is used to dynamically optimize data retrieval operations, allowing the system to efficiently locate and decompress data even when it is partitioned and compressed across multiple nodes. The feedback loop maintains an up-to-date map of data locations and access metadata, reducing the complexity of data access despite the use of partitioning and compression techniques.
Data Source
AI summary
A data input sub-system of a parallelized database system includes processing core resources. Data blocks of a first memory device of a first processing core resource correspond to a first set of logical data block addresses. The processing core resources are operable to obtain divisions of data slabs, compress the divisions of data slabs, and store a respective division of compressed data slabs. A first data slab of a first division of data slabs is mapped to at least a portion of the first set of logical data block addresses that includes at least a portion of a first set of fixed size data fields. The first data slab is compressed to produce a first compressed data slab and the first compressed data slab is mapped to a reduced amount of fixed size data fields of the 10 at least the portion of the first set of fixed size data fields.