Partition Splitting Method for Log-Structured Merge Tree Databases

Solution Overview

Problem

In log-structured merge tree based database systems, partition splitting is resource-intensive due to excessive CPU, IO, and disk space consumption, which can impact performance and online services when large partitions need to be split.

Innovation Solution

A partition splitting method that progressively splits data across multiple rounds, creating new partitions and assigning data based on predetermined rules like range or hash splitting, while deleting split data groups to conserve resources and optimize disk space.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If partition splitting is performed to distribute write/read pressure, then processing capability limit is broken through, but CPU, IO, and disk space resources are excessively consumed

Engineering Contradiction:

Improveprocessing capabilityVSAvoidresource consumption

Core Design Contradiction:

ProductivityVSUse of energy by moving object

Solution Approach 1:

The patent segments the partition splitting process into multiple rounds, where in each round only a subset of data files is split. This divides the large task into smaller manageable units, reducing resource consumption in each iteration while progressively achieving the goal of distributing data across multiple partitions.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent implements periodic partition splitting by performing splitting operations in multiple rounds rather than continuously. Each round processes a portion of data files, and the process repeats until all data files are split. This periodic approach distributes resource consumption over time, preventing excessive peak resource usage while maintaining progress toward the goal.

Inventive Principle:

Principle #19Periodic action

2Productivity

If partition splitting is performed to distribute write/read pressure, then processing capability limit is broken through, but CPU, IO, and disk space resources are excessively consumed

Engineering Contradiction:

Improveprocessing capabilityVSAvoidsplitting time

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The patent segments the partition splitting process into multiple rounds, where in each round only a subset of data files is split. This divides the large task into smaller manageable units, reducing resource consumption in each iteration while progressively achieving the goal of distributing data across multiple partitions.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent implements periodic partition splitting by performing splitting operations in multiple rounds rather than continuously. Each round processes a portion of data files, and the process repeats until all data files are split. This periodic approach distributes resource consumption over time, preventing excessive peak resource usage while maintaining progress toward the goal.

Inventive Principle:

Principle #19Periodic action

3Productivity

If partition splitting is performed to distribute write/read pressure, then processing capability limit is broken through, but online services are impacted

Engineering Contradiction:

Improveprocessing capabilityVSAvoidonline service stability

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The patent segments the partition splitting process into multiple rounds, where in each round only a subset of data files is split. This divides the large task into smaller manageable units, reducing resource consumption in each iteration while progressively achieving the goal of distributing data across multiple partitions.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent implements periodic partition splitting by performing splitting operations in multiple rounds rather than continuously. Each round processes a portion of data files, and the process repeats until all data files are split. This periodic approach distributes resource consumption over time, preventing excessive peak resource usage while maintaining progress toward the goal.

Inventive Principle:

Principle #19Periodic action

Data Source

PatentEP3857402B1Partition splitting method and apparatus, electronic device, and readable storage medium

Publication Date: 2023.03.08 BEIJING OCEANBASE TECHNOLOGY CO LTD

Figure 1~2
Figure 3
Figure 4

AI summary

Implementations of the present application provide a partition splitting method. Data files in a target source partition are divided into at least two data groups. One of the data groups is first used as a target splitting data group, and the target splitting data group is split into a plurality of data sub-groups based on a predetermined splitting rule and the data sub-group is assigned to a corresponding new partition. After the target splitting data group is split, another data group is selected from remaining data groups as a target splitting data group. All the data groups are split in a plurality of rounds of splitting. As such, resource consumption of partition splitting is distributed to a plurality of time periods, thereby effectively saving CPU resources, 10 resources, disk space resources, etc. that are occupied in a splitting process.