Virtual Chunk Spaces and Storage Groups for Node Failure Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed data storage systems face performance bottlenecks in data recovery due to resource-intensive metadata queries and updates when a storage node fails, particularly in systems with billions of data objects, as they need to identify and relocate fragments across multiple nodes efficiently.
Innovation Solution
The implementation of a virtual chunk service (VCS) based storage technique, which assigns unique IDs to storage nodes and forms data storage groups for efficient data fragmentation and recovery, minimizing the need for metadata updates and read operations by using metadata mappings to identify affected data objects and update storage node mappings.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the distributed data storage system uses traditional metadata management to track all data objects across storage nodes, then data reliability is maintained, but the system experiences resource-intensive metadata queries and updates during recovery operations
Solution Approach 1:
The patent segments the metadata management into two distinct components: (1) object metadata that tracks data objects and their fragments, and (2) chunk metadata that tracks storage nodes and their capacity. This segmentation allows the system to query and update only the relevant metadata component during recovery operations, rather than scanning all data object metadata, thereby improving recovery efficiency while maintaining data reliability.
2Reliability
If the system queries metadata for each data object to identify impacted objects during node failure, then complete data recovery is ensured, but the metadata query process becomes resource intensive with billions of data objects
Solution Approach 1:
The patent extracts the tracking of data object locations from the main object metadata and places it in a separate chunk metadata structure. During recovery operations, the system only needs to query the chunk metadata to identify which storage nodes are failed and which chunks are impacted, without having to scan through metadata for all billions of data objects. This extraction significantly reduces the computing resources required while ensuring complete data recovery.
3Reliability
If the system updates metadata for each impacted data object after fragment relocation, then data integrity is maintained, but the metadata update process becomes a performance bottleneck
Solution Approach 1:
The patent segments metadata updates into two separate operations: (1) updating chunk metadata to reflect the new location of relocated chunks, and (2) updating object metadata only for the specific data objects whose fragments were moved. This segmentation eliminates the performance bottleneck by avoiding unnecessary updates to object metadata for data objects whose fragments were not relocated, while still maintaining data integrity through proper tracking of fragment locations.
4Measurement precision
If the system reads all data objects to identify a subset stored on the failed node, then accurate impact identification is achieved, but the process becomes inefficient with billions of data objects
Solution Approach 1:
The patent extracts the location tracking information from the main data object metadata and stores it in a separate chunk metadata structure. This extraction enables the system to identify impacted data objects by querying only the chunk metadata for the failed storage node, rather than reading through all billions of data objects. The accuracy of impact identification is maintained because the chunk metadata contains precise tracking of which chunks are stored on which nodes.
Data Source
AI summary
Technology is disclosed for storing data in a distributed storage system using a virtual chunk service (VCS). In the VCS based storage technique, a storage node (“node”) is split into multiple VCSs and each of the VCSs can be assigned a unique ID in the distributed storage. A set of VCSs from a set of nodes form a storage group, which also can be assigned a unique ID in the distributed storage. When a data object is received for storage, a storage group is identified for the data object, the data object is encoded to generate multiple fragments and each fragment is stored in a VCS of the identified storage group. The data recovery process is made more efficient by using metadata, e.g., VCS to storage node mapping, storage group to VCS mapping, VCS to objects mapping, which eliminates resource intensive read and write operations during recovery.