Virtual Chunk Spaces for Faster Distributed Storage Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed data storage systems face performance bottlenecks in data recovery due to resource-intensive metadata queries and updates when a storage node fails, particularly in systems with billions of data objects, as they need to identify and relocate fragments across multiple nodes efficiently.
Innovation Solution
The Virtual Chunk Service (VCS) based storage technique minimizes computing resources by maintaining metadata that maps VCSs to storage nodes and ECGs, allowing for efficient identification and relocation of data fragments without updating all impacted metadata objects, thus optimizing the data recovery process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional metadata-based data recovery is used in distributed storage systems, then data reliability is maintained, but computational resources and time are excessively consumed due to querying and updating metadata for all data objects
Solution Approach 1:
The patent segments the distributed storage system into multiple storage groups, where each group independently manages its own metadata. This segmentation allows data recovery to be confined to specific storage groups rather than requiring system-wide metadata queries, thereby maintaining data reliability while significantly improving recovery efficiency by limiting the scope of metadata operations to only affected groups.
Solution Approach 2:
The patent introduces a new organizational dimension by layering virtual chunk spaces over physical storage nodes. This creates a hierarchical structure where virtual chunks are mapped to physical locations, enabling recovery operations to target specific virtual chunk spaces without querying metadata for all data objects, thus resolving the contradiction between reliability and recovery efficiency.
2Reliability
If metadata queries are performed for all data objects to identify impacted objects during node failure, then complete data recovery is ensured, but the process becomes resource intensive and slow
Solution Approach 1:
By dividing the storage system into segmented storage groups with isolated metadata, the patent enables recovery operations to be performed only on affected groups rather than querying all data objects system-wide. This ensures complete recovery of impacted data while dramatically reducing the time and resources required by limiting the query scope to relevant segments only.
Solution Approach 2:
The patent performs preliminary organization of data into storage groups with dedicated metadata during the data writing phase. This preliminary action enables the system to quickly identify and recover only the specific storage groups affected by node failures, eliminating the need for time-consuming system-wide metadata queries while ensuring complete recovery of impacted data.
3Reliability
If metadata updates are performed for all impacted data objects after fragment relocation, then data integrity is maintained, but the update process becomes a significant performance bottleneck
Solution Approach 1:
The patent segments metadata management by creating separate metadata spaces for each storage group. When data fragments are relocated, metadata updates are confined to only the affected storage groups rather than requiring updates across all data objects in the system. This maintains data integrity through proper metadata synchronization while significantly reducing update complexity and resource consumption.
Solution Approach 2:
The patent extracts metadata management from a centralized global structure and distributes it across multiple independent storage group metadata spaces. This extraction allows metadata updates to be performed locally within affected storage groups only, maintaining data integrity through proper tracking while eliminating the performance bottleneck of system-wide metadata updates.
4Measurement precision
If the system reads metadata for all data objects to identify a small fraction stored on failed nodes, then accurate recovery targeting is achieved, but the read operation becomes inefficient at scale
Solution Approach 1:
By segmenting the storage system into independent storage groups with dedicated metadata, the patent enables the system to read metadata only from affected storage groups during recovery operations. This maintains accurate identification of recovery targets by confining metadata reads to relevant segments, while dramatically improving read efficiency by avoiding unnecessary reads from unrelated storage groups.
Solution Approach 2:
The patent implements local quality by creating storage-group-specific metadata that is optimized for local recovery operations. Each storage group maintains its own metadata structure that enables precise identification of impacted data objects within that group, achieving accurate recovery target identification while improving read efficiency by accessing only local metadata rather than scanning global metadata for all data objects.
Data Source
AI summary
Technology is disclosed for storing data in a distributed storage system using a virtual chunk service (VCS). In the VCS based storage technique, a storage node (“node”) is split into multiple VCSs and each of the VCSs can be assigned a unique ID in the distributed storage. A set of VCSs from a set of nodes form a storage group, which also can be assigned a unique ID in the distributed storage. When a data object is received for storage, a storage group is identified for the data object, the data object is encoded to generate multiple fragments and each fragment is stored in a VCS of the identified storage group. The data recovery process is made more efficient by using metadata, e.g., VCS to storage node mapping, storage group to VCS mapping, VCS to objects mapping, which eliminates resource intensive read and write operations during recovery.