Logical Memory Partitioning for Parallel Database Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face limitations in processing speed due to hardware constraints, data storage methods, and restricted co-processing options, leading to inefficiencies in data handling and query execution.
Innovation Solution
The implementation of a database system that employs logical partitioning and parallel processing across multiple nodes and processing core resources, utilizing a parallelized data input, storage, and query system, along with advanced data compression techniques like global dictionary compression and run-length encoding, to optimize data storage and query execution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored in a traditional sequential manner in database systems, then hardware resources are utilized, but processing speed is limited due to sequential access constraints
Solution Approach 1:
The patent divides memory into multiple logical partitions (e.g., partition 0, partition 1, partition 2) that can be independently accessed and processed. Each partition can store different data segments, allowing parallel processing operations to occur simultaneously in different memory regions, thereby increasing processing speed while maintaining organized data structures.
Solution Approach 2:
The patent introduces logical partitioning as an additional dimension for organizing memory access. Instead of single-dimensional sequential access, the system creates multi-dimensional access paths through logical partitions, enabling simultaneous access to multiple data segments through different partition indices, thus breaking the sequential access bottleneck.
2Loss of time
If parallel processing is implemented across multiple nodes, then execution time is reduced, but system complexity increases
Solution Approach 1:
The patent divides the database system into multiple processing nodes (node 0, node 1, node 2, etc.) that can execute queries in parallel. Each node maintains independent logical partition structures, allowing simultaneous processing of different data segments without requiring complex inter-node coordination, thus reducing execution time while controlling system complexity.
Solution Approach 2:
The patent creates universal logical partition structures that can be replicated across multiple processing nodes. Each node implements the same logical partitioning mechanism, allowing any node to handle any partition independently. This multi-functional design enables parallel processing without requiring specialized hardware for each node, reducing overall system complexity.
3Productivity
If logical partitioning is implemented in memory, then data access efficiency is improved, but memory management complexity increases
Solution Approach 1:
The patent segments physical memory into multiple logical partitions with distinct indices (partition 0, partition 1, partition 2). Each partition can be independently managed and accessed, allowing efficient data retrieval operations to target specific partitions based on data location. This segmentation improves data access efficiency by reducing the search space while maintaining simple management through partition-index mapping.
Data Source
AI summary
A method includes a host computing device receiving a segment group of data. The method further includes the host computing device evaluating availability status of other computing devices in the storage cluster of computing devices. When one of the other computing devices is unavailable, the method further includes the host computing device dividing the segment group of data into a plurality of lines of data blocks. For a line of the data blocks, the method further includes the host computing device generating at least one parity block. The method further includes the host computing device sending a first data segment that includes first positioned data blocks to a first available computing device. The method further includes the host computing device sending a second data segment that includes second positioned data blocks to a second available computing device. The method further includes the host computing device storing a parity segment.