Replication Engine for Object-Based Storage Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Preserving storage efficiency mechanisms such as deduplication when replicating file system data to object-based storage systems is challenging due to fundamental differences in data layout structure and naming schemas between hierarchical file systems and object-based storage systems.
Innovation Solution
The replication engine maps file system entities into object-based storage objects, preserving deduplication by maintaining primary FBN-to-DBN mappings in indirection objects, and using a parser to identify inode data structures, attribute metadata, and data content, generating inode file objects, indirection objects, and data objects that logically associate file system entities with object-based storage objects.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If file system data is replicated to object-based storage using traditional methods, then data migration is achieved, but storage efficiency mechanisms like deduplication are lost
Solution Approach 1:
The patent introduces an intermediary layer (the replication system with parser and object mapper) that translates between file system entities and OBS objects while preserving deduplication relationships. This intermediary maintains FBN-to-DBN mappings and generates indirection objects that enable deduplication in the OBS environment without requiring changes to the underlying OBS structure.
Solution Approach 2:
The patent changes the parameter representation by mapping file system block numbers (FBNs) to OBS data block numbers (DBNs) and creating indirection objects that preserve the logical relationships. This parameter transformation allows deduplication to be maintained across different storage paradigms by preserving the mapping relationships rather than directly copying data structures.
2Ease of operation
If hierarchical file system structure is converted to flat OBS namespace, then access simplicity is improved, but namespace resolution complexity increases
Solution Approach 1:
The patent segments the namespace resolution process into distinct components: the parser identifies file system entities, the object mapper translates them to OBS objects, and indirection objects provide the mapping layer. This segmentation allows the system to maintain the simplicity of flat OBS access while internally handling the complexity of hierarchical structure translation through separate, specialized components.
3Reliability
If deduplication is preserved during replication, then storage efficiency is maintained, but mapping structure complexity increases
Solution Approach 1:
The patent creates copies of the mapping relationships (FBN-to-DBN mappings) in the form of indirection objects within the OBS environment. Rather than implementing complex real-time deduplication checks, the system copies the essential mapping information needed for deduplication to work effectively in the OBS context, preserving storage efficiency through replicated mapping structures.
Data Source
Figure 1a
Figure 1b
Figure 2
AI summary
A method and system for replicating file system entities in a manner that preserves block-level access and file system efficiency mechanisms such as deduplication are disclosed. In an embodiment, a replication engine receives a stream of file system entities that include a file system inodes and file system data blocks. The replication engine generates object-based storage (OBS) objects based on data and reference information specified by the file system entities. As part of generating the OBS objects, the replication engine generates at least one inode file object that associates file block numbers of a file system inode file and the inode numbers. The replication engine uses inode information to generate reference objects that logically associate file block numbers with data block numbers in per inode manner. The replication engine further generates data objects that contains the file system data blocks and that associates the data blocks with corresponding data block numbers.