Distributed Database Partition Merge Mechanism
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed database systems, partitions created to meet elastic throughput requirements often become underutilized or unused when customer demand decreases, leading to inefficient storage utilization and resource wastage.
Innovation Solution
The method involves merging multiple source partitions into a single 'merged' partition, where data items are copied from source partitions to an intermediate storage structure and then to the merged partition, allowing for the reuse of source partitions and efficient storage space reclamation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple partitions are created to meet elastic throughput requirements, then the system can handle higher customer demand, but storage utilization becomes inefficient when demand decreases
Solution Approach 1:
The system dynamically adjusts the number of active partitions based on customer demand. When demand decreases, previously created partitions can be merged back together, allowing the storage system to adapt its capacity utilization dynamically rather than remaining static at peak capacity levels
Solution Approach 2:
The merge operation recovers storage space by consolidating data from multiple source partitions into a single merged partition. The source partitions are then available to be reused or discarded, effectively recovering the storage resources that were previously allocated but underutilized
2Loss of energy
If partitions are merged to reclaim storage space, then storage utilization improves, but data access speed may be affected during the merging process
Solution Approach 1:
The system performs preliminary actions by creating a merged partition and copying data to it before the merge is complete. This allows the merged partition to be ready and accessible before the source partitions are fully decommissioned, minimizing disruption to data access operations
Solution Approach 2:
The merged partition acts as an intermediary during the transition period. Data can be accessed from the merged partition while the merge operation is in progress, and the merged partition serves as a bridge between the old partition structure and the new consolidated structure, maintaining data availability throughout the process
3Reliability
If data is copied to an intermediate storage structure during merging, then data integrity is maintained, but the merging process complexity increases
Solution Approach 1:
The intermediate storage structure serves as a mediator that simplifies the merging process by providing a temporary holding area for data during the transition. This intermediary structure allows for controlled data movement and validation, making the overall merging process more manageable despite the additional component
Solution Approach 2:
The system creates copies of data in the intermediate storage structure during the merge operation. This copying mechanism ensures data integrity by maintaining redundant copies throughout the transition, allowing for validation and error recovery while the merge progresses
Data Source
AI summary
Partitions are enabled to be merged in storage device of distributed database systems. A request to merge a plurality of source partitions into a merged partition is received. Each source partition stores a plurality of data items having corresponding source global identifiers and corresponding source local identifiers. The data items, with corresponding metadata, are appended from the source partitions to an intermediate storage structure in an order defined by the source local identifiers. Target global identifiers and target local identifiers are generated for each of the data items in the intermediate storage structure. The copied data items are flushed from the intermediate storage structure to the merged partition in an order defined by the target global identifiers. The source partitions and intermediate storage structure may then be purged of data, removed, and/or reused for further storage needs.