De-duplicating Distributed File System Using Cloud Object Store
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed file systems face inefficiencies in de-duplication and storage due to the mismatch between chunk sizes optimized for de-duplication and the larger object sizes that cloud-based object stores are optimized to handle, leading to suboptimal performance and increased overhead.
Innovation Solution
A distributed file system that segments files into chunks of a specific size for de-duplication purposes and stores them as larger objects in cloud-based object stores, using a file system client to compute hashes and send them to a metadata server for deduplication processing, and then combining non-duplicate chunks into larger objects for storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If files are segmented into small chunks for de-duplication, then de-duplication efficiency is improved, but storage overhead increases due to mismatch with cloud object store optimized object sizes
Solution Approach 1:
The patent applies segmentation by dividing files into chunks for de-duplication processing, then further grouping these chunks into larger objects for cloud storage. This two-level segmentation allows small chunk sizes to be used for efficient de-duplication while larger grouped objects are stored in the cloud, resolving the size mismatch problem.
Solution Approach 2:
The patent merges multiple small chunks into larger objects before storing them in the cloud-based object store. This combining process reduces the number of individual storage operations and aligns the stored object sizes with the optimal size ranges for cloud storage systems, thereby reducing storage overhead while preserving de-duplication benefits at the chunk level.
2Measurement precision
If small chunks are used for de-duplication, then de-duplication precision is improved, but the number of storage operations increases leading to reduced performance
Solution Approach 1:
The system segments files into small chunks for de-duplication to achieve high precision in identifying duplicate data. However, it then groups these segments into larger objects for cloud storage, reducing the total number of storage operations and improving overall storage performance.
Solution Approach 2:
The patent performs de-duplication operations on chunks before the final storage operation. By pre-processing the data to identify and eliminate duplicates at the chunk level, the system reduces the amount of data that needs to be stored, thereby reducing the number of actual storage operations required and improving performance.
3Productivity
If large objects are stored in cloud object store, then storage efficiency is improved, but de-duplication effectiveness decreases
Solution Approach 1:
The patent maintains segmentation at the chunk level for de-duplication purposes, allowing effective comparison and identification of duplicates. Simultaneously, it creates larger objects by grouping chunks for cloud storage, achieving both de-duplication effectiveness and storage efficiency.
Solution Approach 2:
The patent operates at two different dimensions: the chunk dimension for de-duplication operations and the object dimension for cloud storage. This dimensional separation allows small chunks to be used for precise de-duplication while larger objects provide storage efficiency in the cloud environment.
4Productivity
If chunk size is optimized for de-duplication, then de-duplication performance is improved, but compatibility with cloud object store optimized object sizes deteriorates
Solution Approach 1:
The system uses segmentation to create chunks optimized for de-duplication performance while maintaining the ability to group these segments into larger objects that are compatible with cloud object store optimizations. This layered approach allows each layer to be optimized for its specific purpose.
Solution Approach 2:
The patent creates a multi-functional system where chunks serve de-duplication purposes and grouped objects serve cloud storage purposes. The same segmented structure supports both de-duplication operations and efficient cloud storage, making the system adaptable to multiple requirements simultaneously.
Data Source
AI summary
Techniques to provide a de-duplicating distributed file system using a cloud-based object store are disclosed. In various embodiments, a request to store a file comprising a plurality of chunks of file data is received. A determination to store at least a subset of the plurality of chunks is made. The request is responded to at least in part by providing an indication to store two or more chunks comprising the at least a subset of the plurality of chunks comprising the file as a single stored object that includes the combined chunk data of said two or more chunks.