Distributed Database Compression for Faster Parallel Query Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database systems face limitations in processing speed due to hardware constraints, data storage methods, and restricted co-processing options, which hinder efficient data processing and query execution.
Innovation Solution
A parallelized database system architecture that divides data into partitions, uses a distributed network of computing devices for parallel processing, and employs global dictionary compression and null elimination techniques to optimize data storage and retrieval, enabling efficient query execution across multiple nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored in traditional database systems with standard hardware, then data storage capacity is maintained, but processing speed is limited by hardware constraints
Solution Approach 1:
The patent segments data into partitions distributed across multiple computing devices, with each device processing specific partitions in parallel. This segmentation allows the system to overcome individual hardware limitations by distributing the processing workload across many devices, thereby improving overall processing speed without requiring any single device to have excessive hardware complexity.
Solution Approach 2:
The patent transitions from a single-machine processing model to a distributed multi-device architecture, adding the dimension of spatial distribution across a network. By organizing computing devices in a distributed network topology and distributing data partitions across them, the system achieves parallel processing capabilities that overcome the processing speed limits of individual hardware units.
2Speed
If data is stored using traditional storage methods, then data accessibility is maintained, but query execution speed is limited by storage retrieval bottlenecks
Solution Approach 1:
The patent divides data into partitions that are distributed across multiple computing devices, with each device maintaining local copies of its assigned partitions. This segmentation enables parallel query execution where multiple devices can simultaneously process different portions of the data, eliminating the single-point retrieval bottleneck of traditional storage methods and improving query execution speed.
Solution Approach 2:
The patent creates distributed copies of data partitions across multiple computing devices in the network. Each device stores and can independently access its local partitions, eliminating the need to retrieve data from a centralized storage system for every query. This copying strategy significantly accelerates query execution by allowing parallel access to data from multiple locations simultaneously.
3Productivity
If co-processing options are restricted, then system simplicity is maintained, but data processing efficiency is limited
Solution Approach 1:
The patent creates a universal distributed processing architecture where each computing device can perform multiple functions: storing data partitions, processing queries locally, and participating in coordinated data processing operations. This multi-functionality enables the system to efficiently handle diverse data processing tasks without requiring specialized co-processing hardware, thereby improving data processing efficiency while maintaining system simplicity.
4Quantity of substance
If data is not compressed, then data accessibility and processing simplicity are maintained, but storage space utilization is inefficient
Solution Approach 1:
The patent applies compression techniques preliminarily during the data partitioning and storage phase, before queries are executed. By pre-compressing data as it is distributed across partitions, the system reduces the storage space required without adding complexity to the query processing operations. The compressed data is then directly accessible for querying, maintaining processing simplicity while achieving efficient storage space utilization.
Data Source
AI summary
A method includes receiving, by a host computing device of a storage cluster of computing devices, a segment group of data. The method further includes processing, by the host computing device, the segment group of data to produce data segments. The method further includes, sending, by the host computing device, the data segments to the computing devices of the storage cluster. The method further includes allocating, by a host node of the first computing device, data segment divisions of the first data segment to nodes of the first computing device. The method further includes allocating, by a host processing core resource of the first node, data segment sub-divisions of the first data segment division to processing core resources of the first node. The method further includes storing, by the first computing device, the first data segment having the data segment divisions and the first data segment division having the data segment sub-divisions.