Dictionary Compression for Parallel Database Query Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database systems face limitations in processing speed due to hardware constraints, data storage methods, and restricted co-processing options, which hinder efficient data processing and query execution.
Innovation Solution
The implementation of a parallelized database system architecture that includes a parallelized data input, query, and storage subsystems, utilizing a distributed network of computing devices with multiple processing core resources for parallel execution of operations, and employing global dictionary compression and data compression techniques to optimize data storage and retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If traditional database storage and processing methods are used, then hardware simplicity is maintained, but processing speed and query execution efficiency deteriorate
Solution Approach 1:
The database system is divided into multiple computing devices forming a distributed network, with each device having multiple processing core resources. Data is partitioned across these devices, enabling parallel processing of queries and data operations, thereby increasing processing speed while distributing system complexity across multiple independent units.
Solution Approach 2:
The system transitions from single-threaded sequential processing to multi-dimensional parallel processing by utilizing multiple processing core resources across multiple computing devices. This dimensional expansion allows simultaneous execution of multiple query operations, fundamentally improving processing speed beyond what single-device sequential processing can achieve.
2Productivity
If data is stored in uncompressed format, then data retrieval and processing simplicity is maintained, but storage efficiency and processing time deteriorate
Solution Approach 1:
Data compression is performed in advance during data input and storage operations. By pre-compressing data before it is fully written to storage, the system reduces the volume of data that needs to be processed and transferred subsequently, thereby improving processing efficiency without adding complexity to the retrieval operations themselves.
Solution Approach 2:
The system dynamically adjusts compression parameters and techniques based on data characteristics and query patterns. By changing compression parameters adaptively, the system optimizes the balance between storage efficiency and processing speed, improving productivity while managing storage complexity through parameter optimization rather than structural complexity.
3Loss of time
If parallel processing is implemented, then query execution speed is improved, but system complexity and coordination overhead increase
Solution Approach 1:
The distributed database system implements self-service mechanisms where each computing device autonomously manages its local processing core resources and makes independent decisions about query execution. This decentralized autonomy reduces the need for complex centralized coordination, allowing parallel processing to proceed with minimal coordination overhead while still achieving reduced query execution times.
Solution Approach 2:
The system introduces intermediary components that facilitate communication and coordination between processing core resources without requiring direct peer-to-peer interaction. These intermediaries manage the complexity of parallel processing coordination, enabling improved query execution time while containing system coordination complexity through standardized intermediary interfaces.
4Productivity
If more processing core resources are added to each computing device, then processing capacity increases, but hardware cost and complexity increase
Solution Approach 1:
Each processing core resource is designed to be universal and multi-functional, capable of executing various types of database operations and queries. This universality allows the system to increase processing capacity by adding identical multi-functional cores rather than requiring specialized hardware for different functions, thereby increasing productivity while controlling hardware complexity through standardization.
Data Source
AI summary
A method includes receiving a data set that includes a plurality of data records, where a data record includes a first data field containing a first fixed length data value and a second data field containing a first variable length data value. The method further includes accessing a compression dictionary for the second data field, where a first entry of the compression dictionary includes a key field storing a first fixed length index value and a value field storing the first variable length data value, and where the key field has a smaller data size than the value field. The method further includes creating a storage data set based on the compression dictionary and sending the storage data set to a storage sub-system for storage, where the first variable length data value of the second data field of the data record is replaced with the first fixed length index value.