Dynamic Adaptive Partition Splitting in Distributed Databases
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing partition splitting techniques in multi-partitioned distributed databases often result in restricted access and prolonged downtime due to the need to maintain data consistency, which is inconvenient for users and affects availability.
Innovation Solution
Implementing an asynchronous pre-split optimization phase that processes existing records in the background using a snapshot of the partition, allowing writes to continue, and employing a low-collision non-cryptographically secure hashing algorithm like FarmHash Fingerprint64 to speed up record processing, thereby reducing the time window for write blocking during partition splits.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If partition splitting is performed to improve data access and control partition size, then storage scalability is improved, but access to the partition is restricted during processing which affects availability
Solution Approach 1:
The system creates new partition structures and computes initial metrics schemas in advance before the split is finalized. By preparing the second and third partitions with their metric schemas beforehand and only blocking writes during the final metrics computation phase, the system reduces the duration of access restriction while ensuring data consistency.
Solution Approach 2:
The partition splitting process is divided into distinct phases: creating new partitions with initial schemas, computing metrics from snapshots in the background, and finally applying split update commands. This segmentation allows different parts of the system to operate independently at different stages, minimizing the impact on overall availability.
2Ease of operation
If partition splitting is performed to improve data access, then data accessibility is improved, but downtime is prolonged due to write blocking
Solution Approach 1:
The system takes snapshots of the first partition and creates the second and third partitions with their metric schemas before blocking writes. This preliminary preparation ensures that when the split finalizes, the new partitions are already structured and ready, minimizing the time writes are blocked.
Solution Approach 2:
The system continues to accept and process write operations to the first partition during the partition splitting process by maintaining the first partition's writability. Write operations continue uninterrupted while the system simultaneously computes metrics from snapshots and prepares new partitions, ensuring continuous useful action without downtime.
3Reliability
If metrics schemas are computed from partition snapshots to ensure data consistency, then data consistency is maintained, but the processing time increases
Solution Approach 1:
The system computes metrics schemas from snapshots in advance before the partition split is finalized. By performing this computation beforehand using background processing, the system ensures data consistency is established prior to the split, reducing the time writes are blocked during the finalization phase.
Solution Approach 2:
The system uses snapshots as an intermediary representation of the partition state. Instead of computing metrics directly from the live partition during writes, the snapshot serves as a stable intermediate state that can be processed independently, ensuring consistency without blocking writes to the original partition.
4Speed
If non-cryptographically secure hashing algorithm is used to speed up record processing, then processing speed is improved, but security is reduced
Solution Approach 1:
The system applies different hashing algorithms to different parts of the system based on their security requirements. Non-cryptographically secure hashing (FarmHash Fingerprint64) is used for internal record processing and metrics computation where security is not critical and speed is essential, while cryptographically secure hashing is used for security-sensitive operations like authentication and data integrity verification.
Data Source
AI summary
In some examples, a computing device may store a first snapshot of a state of data in a first partition at a first point in time. The computing device may create a second partition and a third partition to each receive a portion of the data, the second partition and the third partition each including a metrics schema, and may determine information for the metrics schemas based on information in the first snapshot. During the determining of the information for the metrics schemas, the computing device may receive a write to the first partition. The computing device may update the first partition based on the write and may add a split update command to a data structure based on the write. In addition, the computing device may update at least one of the metrics schemas in the second partition or the third partition based on the split update command.