Virtual File Reconstruction from Distributed Storage Tiers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current deduplication systems face challenges in recovering corrupt files when primary and secondary copies are lost, as they rely on backup solutions that may not have the necessary data to restore the files, leading to potential permanent data loss.
Innovation Solution
A method and system for reconstructing corrupt files by generating a virtual user file from available nodes or tiers, using fingerprint signatures and Merkle tree representations to identify and recreate missing segments, allowing for data recovery across different storage tiers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is moved from active tier to cloud tier for long-term storage, then storage cost is reduced and data retention is improved, but data accessibility and recovery capability are worsened
Solution Approach 1:
The patent segments data into multiple chunks distributed across different storage tiers (active and cloud). Each chunk is independently stored and can be individually recovered. This segmentation allows the system to maintain data on cost-effective cloud storage while preserving recovery capability through distributed chunk placement across tiers.
Solution Approach 2:
The patent introduces an intermediary recovery mechanism that acts as a mediator between cloud storage and active storage. When data recovery is needed, the system uses intermediary processes to retrieve chunks from cloud storage and reconstruct original files, bridging the gap between long-term archival storage and immediate data accessibility.
2Quantity of substance
If deduplication is implemented to reduce storage overhead, then storage efficiency is improved, but processing overhead and system complexity are worsened
Solution Approach 1:
The patent implements a universal deduplication mechanism that operates across multiple storage tiers simultaneously. The same deduplication logic and fingerprinting process is applied whether data is stored on active tier or cloud tier, creating a unified system that handles deduplication consistently across different storage environments without requiring separate complex systems for each tier.
Solution Approach 2:
The patent uses fingerprint signatures as copies of data characteristics to identify duplicates. Instead of copying actual data blocks for comparison, the system creates and compares compact fingerprint representations, significantly reducing the processing overhead and memory requirements while maintaining accurate duplicate detection across distributed storage tiers.
3Reliability
If backup copies are stored in separate locations for data protection, then data security is improved, but data integrity and complete recovery are worsened when backup copies are also lost
Solution Approach 1:
The patent merges multiple storage tiers (active and cloud) into a unified data protection ecosystem. Data chunks are distributed across both tiers, and the system combines resources from both locations to reconstruct lost files. This merging ensures that even if one tier fails or loses data, the other tier contains sufficient chunks to enable complete file recovery.
Data Source
AI summary
Embodiments of method and system are directed to a best effort method to reconstruct the corrupt files from available nodes or tiers and heal or repair the corrupt files. The method looks for the existence of fingerprints of the lost segments in available tiers on the nodes or across nodes within a cluster. The method discovers the copies of these missing segments, creates an Lp tree referring these discovered segments and creates a virtual file in the namespace. The file is then moved to the destination tier where the corruption exists, thus healing the corruption.