Storage De-duplication via Content-Based Address Translation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage systems face inefficiencies in de-duplication across a cluster of interconnected storage systems, as data access requests are directed to different systems based on file offsets rather than content, leading to compromised de-duplication efficiency and increased storage resource consumption.
Innovation Solution
Implementing a system that uses data container offset indexing in combination with data content redirection, where the Locate by offset function determines the storage server responsible for a file region and the Locate by content function ensures data is stored based on content, allowing identical blocks to be served by the same server for de-duplication purposes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data access requests are directed to storage systems based on file offsets, then data distribution across the cluster is achieved, but de-duplication efficiency is compromised
Solution Approach 1:
The patent introduces a content-based address translation layer that acts as an intermediary between the offset-based striping mechanism and the actual data storage. This translation layer maps logical offsets to physical locations based on content hash, enabling both distributed access and effective de-duplication without direct conflict between the two mechanisms
Solution Approach 2:
The system dynamically changes the addressing parameter from offset-based to content-hash-based for de-duplication operations. By switching the routing criterion based on the operation type (access vs. de-duplication), the system optimizes both data distribution and deduplication efficiency
2Reliability
If multiple copies of identical data are stored across different storage systems, then data availability is improved, but storage space consumption increases
Solution Approach 1:
The patent merges multiple copies of identical data into a single canonical copy by using content-based addressing. When identical data is detected through hash matching, subsequent writes are redirected to the existing canonical location rather than creating new copies, thereby reducing storage consumption while maintaining availability through the distributed architecture
Solution Approach 2:
The system discards redundant duplicate copies and recovers storage space by redirecting future writes to the canonical copy. The metadata system tracks the canonical location, allowing the system to eliminate wasted storage while preserving data availability through the distributed file system architecture
3Productivity
If data is stored on different storage systems based on offset, then load distribution is achieved, but cache memory performance deteriorates
Solution Approach 1:
The content-based address translation layer serves as an intermediary that enables cache optimization by identifying identical data blocks across the cluster. This layer allows the cache system to prefetch and cache duplicate blocks from different storage systems, improving access speed while maintaining load distribution through the underlying offset-based striping
Data Source
AI summary
A system and method enables de-duplication in a storage system architecture comprising one or more volumes distributed across a plurality of nodes interconnected as a cluster. De-duplication is enabled through the use of file offset indexing in combination with data content redirection. File offset indexing is illustratively embodied as a Locate by offset function, while data content redirection is embodied as a novel Locate by content function. In response to input of, inter alia, a data container (file) offset, the Locate by offset function returns a data container (file) index that is used to determine a storage server that is responsible for a particular region of the file. The Locate by content function is then invoked to determine the storage server that actually stores the requested data on disk. Notably, the content function ensures that data is stored on a volume of a storage server based on the content of that data rather than based on its offset within a file. This aspect of the invention ensures that all blocks having identical data content are served by the same storage server so that it may implement de-duplication to conserve storage space on disk and increase cache efficiency of memory.