Virtual Chunk Spaces and Storage Groups for Node Failure Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed data storage systems face performance bottlenecks in data recovery due to resource-intensive metadata queries and updates when a storage node fails, particularly in systems with billions of data objects, as they need to identify and relocate fragments across multiple nodes efficiently.
Innovation Solution
The Virtual Chunk Service (VCS) based storage technique minimizes computing resources by using unique IDs for VCSs and storage groups, allowing for efficient data recovery by updating only the VCS storage node mapping and eliminating the need to read or update metadata for all impacted data objects, thus optimizing the recovery process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the distributed data storage system uses traditional metadata-based data recovery methods, then data reliability is maintained, but the recovery process becomes resource intensive and slow due to metadata queries and updates for each data object
Solution Approach 1:
The patent segments the storage system into storage groups, where each storage group contains a subset of storage nodes and associated data objects. This segmentation allows the system to isolate and repair only the affected storage group when a node fails, rather than processing metadata for all data objects in the entire system. The segmentation principle directly resolves the contradiction by maintaining reliability through structured organization while improving productivity by limiting the scope of recovery operations.
Solution Approach 2:
The patent implements preliminary action by pre-organizing data objects into storage groups and pre-establishing the relationships between storage nodes and data objects within each group. This preliminary structuring enables the system to quickly identify and repair only the relevant storage group when a failure occurs, without needing to query metadata for all data objects. The preliminary organization maintains reliability while significantly improving recovery efficiency.
2Reliability
If the system queries metadata for each data object to identify impacted objects during node failure, then complete data recovery is ensured, but the process becomes inefficient due to the significantly large number of data objects (billions)
Solution Approach 1:
By segmenting the system into storage groups with defined boundaries, the patent eliminates the need to query metadata for all billions of data objects during recovery. When a storage node fails, the system only needs to identify and process the relatively small subset of data objects within the affected storage group. This segmentation maintains recovery completeness while dramatically reducing the time required.
Solution Approach 2:
The patent applies partial action by performing recovery operations only on the necessary subset of data objects within the affected storage group, rather than processing all data objects in the system. This partial approach ensures that complete recovery of impacted data is achieved while avoiding the excessive time cost of processing the entire dataset.
3Reliability
If the system updates metadata for each impacted data object after fragment relocation, then data integrity is maintained, but the resource consumption and processing overhead increase significantly
Solution Approach 1:
The patent segments the metadata update operation to only affect the storage group level rather than individual data object level. When fragments are relocated within a storage group, the system updates the storage group metadata to reflect the new fragment locations, without needing to update metadata for each individual data object. This segmentation maintains data integrity while significantly reducing computational resource consumption.
Solution Approach 2:
The patent merges multiple individual metadata update operations into a single storage group-level metadata update. Instead of updating metadata for each data object whose fragments were relocated, the system performs a consolidated update at the storage group level, maintaining data integrity while reducing the overall computational overhead and energy consumption.
Data Source
AI summary
Technology is disclosed for storing data in a distributed storage system using a virtual chunk service (VCS). In the VCS based storage technique, a storage node (“node”) is split into multiple VCSs and each of the VCSs can be assigned a unique ID in the distributed storage. A set of VCSs from a set of nodes form a storage group, which also can be assigned a unique ID in the distributed storage. When a data object is received for storage, a storage group is identified for the data object, the data object is encoded to generate multiple fragments and each fragment is stored in a VCS of the identified storage group. The data recovery process is made more efficient by using metadata, e.g., VCS to storage node mapping, storage group to VCS mapping, VCS to objects mapping, which eliminates resource intensive read and write operations during recovery.