Database Partition Metadata Merging to Reduce I/O Overhead
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed database systems, the process of merging adjacent partitions is inefficient due to the need to read and write large quantities of data, resulting in significant overhead and prolonged partition merging times.
Innovation Solution
A method where the metadata of two adjacent partitions is merged to create metadata for a new partition, allowing data from the original partitions to be accessed through the new metadata, thereby reducing the need for extensive data read/write operations and speeding up the partition merging process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data entries in one partition are read by traversal and written into the other partition during merging, then the partition merging can be completed, but data read/write operations increase greatly and overheads are greatly increased
Solution Approach 1:
The patent segments the partition merging process into two distinct phases: metadata merging phase (where only metadata is merged without data read/write) and data access phase (where data is accessed through the new merged metadata). This segmentation allows the system to avoid full data traversal during merging while still achieving complete partition consolidation, thereby resolving the contradiction between merging speed and data read/write time
Solution Approach 2:
The patent performs preliminary metadata merging before actual data access is needed. By merging metadata first and establishing the merged partition structure in advance, the system prepares the access pathway beforehand, allowing subsequent data operations to proceed efficiently without requiring extensive data read/write during the merging process itself
2Reliability
If service write operations on the two partitions are temporarily frozen until partition merging is completed, then data consistency is maintained, but the time of freezing the service write operations is prolonged
Solution Approach 1:
The patent segments the service write freezing into two stages: partial freezing during metadata merging (affecting only metadata operations) and complete freezing only when necessary for data consistency. This segmentation reduces the overall freezing time while maintaining data consistency by carefully controlling when freezes are applied
Solution Approach 2:
The patent performs preliminary metadata merging with minimal or no service write freezing, establishing the merged partition structure before full data consistency checks are required. This preliminary action allows the system to prepare the merged state in advance, reducing the duration of complete service write freezing needed later
Data Source
AI summary
A partition merging method includes obtaining, by a first database server, first metadata of a first partition based on an identifier of a current file of the first partition, obtaining second metadata of a second partition based on an identifier of a current file of the second partition, and generating third metadata of a third partition based on the first metadata and the second metadata.