Partition Balancing Subtask Issuing for Distributed Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed storage systems, the long duration of partition balancing tasks leads to unreliable data recovery due to the inability to execute subsequent tasks before the previous ones complete, resulting in a decrease in system reliability.
Innovation Solution
A method and apparatus for issuing partition balancing subtasks that allow concurrent processing of preceding and succeeding tasks by generating and managing subtasks to migrate partitions from faulty nodes, calculating target distribution information, and executing data migration tasks, while deleting incomplete and repeated subtasks to optimize data distribution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If partition balancing tasks are executed sequentially (waiting for previous tasks to complete), then data distribution accuracy is improved, but system reliability decreases and recovery time increases
Solution Approach 1:
The partition balancing task is segmented into multiple subtasks that can be executed concurrently. Each subtask handles a specific portion of the data migration process, allowing parallel execution while maintaining overall data distribution accuracy through coordinated management of subtasks.
Solution Approach 2:
The system performs preliminary actions by pre-calculating target partition distribution information and preparing migration plans before actual data migration begins. This allows the system to establish a roadmap for concurrent subtask execution while ensuring the final data distribution meets accuracy requirements.
2Manufacturing precision
If partition balancing tasks are executed sequentially, then data distribution accuracy is improved, but data recovery time increases
Solution Approach 1:
The partition balancing task is divided into multiple independent subtasks that can run in parallel. Each subtask processes a specific data migration operation, enabling concurrent execution that significantly reduces total recovery time while maintaining data distribution accuracy through coordinated subtask management.
Solution Approach 2:
The system maintains continuous useful action by executing multiple partition balancing subtasks concurrently without idle waiting periods. While previous approaches required sequential completion, this system continuously processes multiple migration operations in parallel, maximizing resource utilization and reducing overall recovery time.
3Reliability
If multiple partition balancing tasks are issued concurrently, then system reliability and productivity are improved, but task management complexity increases
Solution Approach 1:
The complex partition balancing task is segmented into smaller, manageable subtasks with clearly defined responsibilities. This segmentation simplifies management by making each subtask independent and easier to track, while the overall system achieves higher reliability through concurrent execution of multiple simplified subtasks.
Solution Approach 2:
The system introduces an intermediary mechanism (the partition management module) that coordinates between multiple concurrent subtasks and the underlying storage system. This intermediary manages task scheduling, monitors execution status, and handles conflicts, thereby reducing the apparent complexity for external systems while enabling concurrent operation.
4Productivity
If multiple partition balancing tasks are issued concurrently, then productivity is improved, but task management complexity increases
Solution Approach 1:
The partition balancing operation is segmented into multiple parallelizable subtasks, each handling specific data migration operations. This segmentation enables concurrent execution that boosts productivity while keeping individual subtasks simple enough to manage efficiently through standardized interfaces and coordination mechanisms.
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
The present invention provides a method, an apparatus, and a system for issuing a partition balancing subtask, which are applied to a controller. The controller and at least two nodes form a storage system. After receiving a second partition balancing task, the controller generates a second partition balancing subtask set, where the second partition balancing subtask set includes at least one partition balancing subtask, and each partition balancing subtask records: a migration partition, a node to which the migration partition belongs, and a destination node; searches a current partition balancing subtask set, and deletes a repeated partition balancing subtask between the second partition balancing subtask set and the current partition balancing subtask set; and issues remaining partition balancing subtasks after the repeated partition balancing subtask is deleted to the destination node recorded in each partition balancing subtask. The present invention may be used to implement migration of a partition between nodes.