Distributed Data Repair via Segmented Range Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed systems face inefficiencies in data repair due to repeated repairs on multiple nodes, leading to resource waste and high maintenance costs, particularly in systems like Cassandra and Scylladb, where data consistency across multiple copies is challenging to maintain without causing redundant calculations and resource consumption spikes.
Innovation Solution
The method involves segmenting master data ranges into finer sub-data ranges, generating and prioritizing repair tasks, and executing them in a controlled manner to ensure data consistency between master and copy data, allowing for breakpoint resume and efficient resource management by avoiding instantaneous resource spikes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data repair is performed on multiple nodes to ensure data consistency, then data reliability is improved, but system resource overhead and operation maintenance costs increase
Solution Approach 1:
The patent segments the data repair process into two distinct phases: a fast repair phase that quickly repairs data on the current node, and a slow repair phase that repairs data on other nodes. This segmentation allows the system to prioritize local repairs for immediate consistency while deferring remote repairs to reduce peak resource consumption, thereby resolving the contradiction between ensuring data consistency and minimizing system resource overhead.
2Productivity
If data repair is performed on all nodes simultaneously, then data consistency is achieved faster, but resource consumption spikes instantaneously
Solution Approach 1:
The patent implements periodic action by dividing data repair into sequential phases (fast repair phase followed by slow repair phase) rather than executing all repairs simultaneously. The fast repair phase completes critical repairs first, then the slow repair phase continues with remaining repairs at a controlled pace. This periodic approach maintains productivity by ensuring repairs are completed while avoiding instantaneous resource consumption spikes through phased execution.
3Reliability
If repair tasks are executed without prioritization, then all repair operations are performed, but task management complexity increases
Solution Approach 1:
The patent applies local quality by assigning different priorities to different repair tasks based on their location and importance. Fast repair tasks on the current node are given high priority for immediate execution, while slow repair tasks on other nodes are given lower priority. This prioritization strategy ensures critical repairs are completed first while simplifying task management through a clear hierarchy, resolving the contradiction between achieving data consistency and managing task complexity.
Data Source
AI summary
A data processing method and apparatus, and an electronic device, and a storage medium. The method comprises: determining a master data range on the current node (S102), master data within the master data range corresponds to multiple pieces of copy data stored on other nodes; segmenting the master data range into multiple first sub-data ranges (S104); and performing data recovery on each of the first sub-data ranges, so as to repair inconsistent data between sub-data in the first sub-data ranges and corresponding copy sub-data in the copy data to make them consistent. The technical solutions of the present disclosure overcome the defect of resource waste caused by performing repeated repair on data with multiple copies stored on multiple nodes during a data repair process.