Data and Parity Section Layout for Parallel Query Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database systems face limitations in processing speed due to hardware constraints, data storage methods, and restricted co-processing options, which hinder efficient data processing and query execution.
Innovation Solution
The implementation of a parallelized database system architecture that includes a parallelized data input, query, and storage subsystems, utilizing distributed data partitions, redundancy encoding, and independent processing core resources to optimize data storage and retrieval processes, allowing for lock-free and parallel execution of operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is stored in a unified storage structure, then storage simplicity is maintained, but processing speed and query efficiency deteriorate due to hardware constraints and data access bottlenecks
Solution Approach 1:
The patent divides the storage structure into separate data storage sections and parity storage sections. Data blocks are stored in data sections while corresponding parity blocks are stored in parity sections, allowing independent access and processing of data and parity information, thereby improving processing speed without excessive complexity
Solution Approach 2:
The patent extracts parity blocks from the unified storage structure and places them in dedicated parity storage sections. This separation allows the data processing subsystem to access data blocks without being blocked by parity block access, improving query efficiency and processing speed
2Productivity
If distributed data partitions are used, then processing parallelism is improved, but data access coordination complexity increases
Solution Approach 1:
The patent divides data into multiple data partitions distributed across different computing devices. Each partition can be processed independently by different processing core resources, enabling parallel processing that improves productivity while the segmentation itself helps manage coordination complexity
Solution Approach 2:
The patent implements lock-free data structures and operations that allow multiple processing core resources to access and modify data partitions simultaneously without requiring complex coordination or locking mechanisms. Each processing core can independently access data blocks and parity blocks without interfering with other cores, enabling parallel processing with minimal coordination overhead
3Reliability
If redundancy encoding is implemented, then data reliability is improved, but storage space requirements increase
Solution Approach 1:
The patent extracts parity blocks from the data storage sections and stores them in separate parity storage sections. This allows the system to maintain full redundancy encoding capability while freeing up data storage space for actual data, effectively managing the trade-off between reliability and storage space
Solution Approach 2:
The patent applies redundancy encoding locally to each data partition rather than globally across all data. This allows the system to maintain data reliability through parity blocks while limiting the storage overhead to only what is necessary for each local partition, improving overall storage efficiency
Data Source
AI summary
A method includes generating, by a processing entity of a computing system, a plurality of parity blocks from a plurality of lines of data blocks. A first number of parity blocks of the plurality of parity blocks is generated from a first line of data blocks of the plurality of lines of data blocks. The method further includes storing, by the processing entity, the plurality of lines of data blocks in data sections of memory of a cluster of computing devices of the computing system in accordance with a read/write balancing pattern and a restricted file system. The method further includes storing, by the processing entity, the plurality of parity blocks in parity sections of memory of the cluster of computing devices in accordance with the read/write balancing pattern and the restricted file system.