Distributed Storage Read Layout Across Maintenance Domains
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems face challenges in maintaining data accessibility during maintenance or system failures, leading to impaired user operations due to data unavailability in data centers.
Innovation Solution
A distributed storage system architecture that employs single-sided operations with RDMA-capable network interface controllers, combined with Reed-Solomon and layered/nested coding techniques, to ensure data redundancy and enable reconstruction of lost or damaged data chunks across multiple maintenance domains.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored in a distributed storage system with maintenance domains, then system reliability is improved, but data accessibility during maintenance deteriorates
Solution Approach 1:
The storage system is segmented into multiple maintenance domains, each independently manageable. Data chunks are distributed across these domains, allowing maintenance operations to be performed on individual domains without affecting the entire system. This segmentation enables the system to maintain reliability while preserving data accessibility through parallel operations on active domains.
Solution Approach 2:
The system dynamically changes the operational state parameters of maintenance domains, transitioning them between active and inactive states based on maintenance requirements. By adjusting which domains are active versus inactive, the system can perform maintenance on specific domains while maintaining data accessibility through other active domains, thus resolving the contradiction between reliability improvement and data accessibility.
2Reliability
If data chunks are distributed across multiple maintenance domains, then system availability is improved, but system complexity increases
Solution Approach 1:
The system divides the storage infrastructure into discrete maintenance domains with clear boundaries and responsibilities. Each domain manages its own data chunks independently, which simplifies the overall system architecture by creating modular, manageable units. This segmentation reduces complexity while maintaining availability through the distributed nature of the domains.
Solution Approach 2:
The system performs preliminary organization of data chunks into specific maintenance domains before maintenance operations begin. This pre-arranged distribution structure is established in advance, creating a clear mapping between data and domains that simplifies maintenance planning and execution. The preliminary action reduces operational complexity during actual maintenance events while preserving system availability.
Data Source
Figure 1A
Figure 1B
Figure 2
AI summary
A method (800) of distributing data (312) in a distributed storage system (100) includes receiving a file (310) into non-transitory memory (204) and dividing the received file into chunks (330). The chunks are data-chunks (330nD) and non-data chunks (330nC). The method also includes grouping one or more of the data chunks and one or more of the non-data chunks in a group (G). One or more chunks of the group is/are capable of being reconstructed from other chunks of the group. The method includes distributing the chunks of the group to storage devices (114) of the distributed storage system based on a distributed storage system hierarchy (400). The hierarchy includes maintenance domains (402) having active and inactive states, each storage device associated with a maintenance domain, the chunks of a group are distributed across multiple maintenance domains to maintain the ability to reconstruct chunks of the group.