Deduplication Engine Random Access via Dual-Store Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Virtual Tape Libraries (VTL) do not enable easy access to stored data using existing file share systems like NFS and CIFS due to their sequential data storage model, which restricts random IO access and deduplication efficiency.
Innovation Solution
A data deduplication apparatus with a controller and storage facility that includes a fast memory for random access, a VTL interface for emulating physical tape libraries, and a NAS interface for handling Network Attached Storage protocols, allowing for inline deduplication and random access while maintaining data integrity through CRC calculations and bucketization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is stored in a sequential manner in VTL, then storage capacity and deduplication efficiency are improved, but random IO access capability deteriorates
Solution Approach 1:
The patent segments the storage system into two distinct stores: a first store for sequential storage of deduplicated data and a second store for random access of file metadata and data portions. This segmentation allows each store to be optimized for its specific function, resolving the contradiction between sequential storage efficiency and random access capability.
Solution Approach 2:
The patent introduces a file system layer as an intermediary between the sequential store and users. This intermediary translates random IO requests into sequential access patterns by retrieving data in contiguous blocks, thereby enabling random access semantics while maintaining sequential physical storage benefits.
2Reliability
If VTL emulates physical tape libraries, then data integrity and sequential access are maintained, but compatibility with existing file share systems deteriorates
Solution Approach 1:
The patent implements multiple interfaces including VTL interface for tape emulation and NAS interface for network file access, allowing the same underlying storage system to serve multiple functions and be compatible with different access paradigms simultaneously.
Solution Approach 2:
The system dynamically routes access requests through different interfaces based on the type of access needed, with the file system layer adapting its behavior to provide appropriate access patterns while maintaining data integrity across all access methods.
3Quantity of substance
If inline deduplication is implemented, then storage capacity is improved, but processing overhead and complexity increase
Solution Approach 1:
The patent performs deduplication as a preliminary action during the data ingestion phase rather than during retrieval. Data is deduplicated at the time of write, and subsequent access operations simply retrieve the deduplicated data without requiring additional processing, thereby reducing overall system complexity.
Data Source
AI summary
A deduplication engine is operable to select at least two chunks of data for deduplication and deduplicate the selected at least two chunks of data. A first store is operable to store the deduplicated chunks of data in a sequential manner, and a second store is operable to store at least a portion of at least one chunk of the deduplicated data in a manner to allow random access, where data is accessed via the first and/or second store.