Hierarchical Metadata for Streaming Data Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems, such as DELL EMC's PRAVEGA, face challenges in data recovery after system failures, particularly when data streams are not replicated, leading to loss of event data stored in Tier-2 storage, which becomes unreadable even after system restart.
Innovation Solution
Implementing a hierarchical naming system and metadata association with data chunks in Tier-2 storage, allowing for the recreation of epochs and segments, and reordering of chunks to restore the data stream, enabling recovery of the major part of the data stream even after system failure.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If data is stored in Tier-2 storage without replication, then storage cost is reduced, but data reliability deteriorates causing complete data loss after system failure
Solution Approach 1:
The patent applies preliminary action by embedding hierarchical metadata (epoch information, segment identifiers, chunk ordering) into the data structure before storage. This metadata is organized in a hierarchical naming system that enables automatic reconstruction of the data stream without requiring replication or external recovery mechanisms. When failure occurs, the pre-organized metadata guides the recovery process to reconstruct epochs and segments in the correct order.
2Device complexity
If data streams are not replicated, then system complexity is reduced, but data recoverability deteriorates making raw data unreadable after failure
Solution Approach 1:
The patent applies segmentation by dividing the data stream into discrete epochs and segments with hierarchical organization. Each chunk contains metadata identifying its position in the hierarchy (epoch number, segment identifier). This segmentation allows the system to recover data in organized units without requiring complex replication, as each segment can be independently identified and reassembled using its embedded metadata.
Solution Approach 2:
The patent introduces hierarchical metadata as an intermediary between the raw data and the recovery process. This metadata layer contains epoch information, segment identifiers, and ordering information that mediates the reconstruction process. Instead of requiring complex replication logic, the metadata acts as a guide that enables automatic reconstruction of the data stream hierarchy from scattered chunks.
3Reliability
If hierarchical metadata structure is implemented, then data recovery capability is improved, but storage overhead increases
Solution Approach 1:
The patent merges the metadata with the data chunks themselves rather than storing metadata separately. The hierarchical naming system encodes epoch and segment information directly in the chunk identifiers and file paths. This merging eliminates the need for separate metadata storage structures, reducing overall storage overhead while maintaining full recovery capability. The metadata is embedded in the naming convention and chunk structure rather than requiring additional storage resources.
Data Source
AI summary
A streaming data storage system maintains a hierarchical structure of metadata in association with the data chunks of streams, in which the metadata facilitates recovery of the data streams if the streaming data storage system fails. In one implementation, the metadata comprises the pathnames and filenames of the chunks stored in a file storage system or object storage system, in which each pathname represents the epoch and segment of the chunks in the segment, and the chunks' filenames represent the relative ordering of the chunks in the segment. To recover the data stream, the epochs and their segments are recreated, and the segments are repopulated with references to their respective chunks. Once recovered, a new epoch is created with a number of active segments equal to the number of segments of the last recreated epoch, and event appends to the data stream can resume.