Meta-Copyset Storage Layout for Fault-Tolerant Data Migration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current storage systems face challenges with data unavailability due to limited copysets, where all machines in a copyset becoming unavailable lead to significant data unavailability, and erasure coding systems are inefficient in handling simultaneous machine failures, necessitating improved fault-tolerant and efficient data storage solutions.
Innovation Solution
The implementation of meta-copysets, which are sets of machines from which multiple copysets are generated, allowing for efficient data migration and increased storage-eligible copysets while maintaining low data unavailability risks, supports both replication-based and erasure-coded storage by reusing machines and minimizing failure correlation among copysets within the same meta-copyset.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a limited number of copysets are used to reduce the probability of simultaneous machine unavailability, then data unavailability risk is reduced, but the system cannot efficiently handle migration when all machines in a copyset become unavailable
Solution Approach 1:
The patent divides the storage system into a hierarchical structure with multiple meta-copysets, each containing multiple copysets. This segmentation allows the system to maintain limited copysets within each meta-copyset for reliability while providing multiple meta-copysets for migration capability when failures occur.
Solution Approach 2:
The patent introduces a new dimensional organization by creating meta-copysets that group multiple copysets together. This adds a hierarchical dimension to the traditional flat copyset structure, enabling migration across meta-copysets while maintaining the benefits of limited copysets within each meta-copyset.
2Ease of operation
If data is stored in a random copyset, then distribution is simple, but the probability that all machines of at least one copyset become unavailable increases
Solution Approach 1:
The patent segments the copyset space into multiple meta-copysets, each with its own set of copysets. Data is assigned to random copysets within a meta-copyset, maintaining simplicity while the meta-copyset structure provides redundancy against complete failure of any single copyset group.
Solution Approach 2:
The meta-copyset structure provides beforehand cushioning by ensuring that data has backup locations across different meta-copysets. If all machines in one copyset become unavailable, the data can be recovered from other copysets within the same or different meta-copysets.
3Quantity of substance
If erasure coding is used to improve storage efficiency, then storage capacity utilization increases, but data becomes unavailable when several machines are unavailable simultaneously
Solution Approach 1:
The patent applies segmentation by organizing erasure-coded data across multiple meta-copysets with multiple copysets each. This ensures that the required number of fragments for reconstruction are distributed across different failure domains, maintaining availability even when several machines fail simultaneously.
Solution Approach 2:
The patent adds a hierarchical dimension to erasure coding by organizing fragments across meta-copysets. This multi-dimensional distribution ensures that erasure coding thresholds are met even when failures occur within specific copysets or meta-copysets, improving reliability while maintaining storage efficiency.
Data Source
AI summary
A system includes determination of a plurality of meta-copysets of a plurality of storage devices, each of the meta-copysets including a first number of storage devices, determination of a first copyset of a second number of storage devices from a first one of the meta-copysets, where the second number is less than the first number, storage of first data in a fault-tolerant format using the first copyset, determination to migrate the first data from a first storage device of the first copyset, and, in response to the determination to migrate the first data, determine a second copyset of the second number of storage devices including a storage device from the first meta-copyset which is not in the first copyset, and the storage devices of the first copyset except for the first storage device of the first copyset and storage of the first data in a fault-tolerant format using the second copyset.