Global Deduplication Engine for Multi-Node Storage Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional Data Domain File System (DDFS) virtual scaled out architecture only deduplicates files within a single data node, leading to reduced storage efficiency in multi-node systems compared to single node architectures.
Innovation Solution
A method and system for redistributing files across multiple data nodes based on similarity indices, using a Global Deduplication Engine (GDE) to generate ranked recipes for file movement, optimizing deduplication space savings by placing similar files on the same node, while adhering to storage size limits and minimizing data movement costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If files are deduplicated only within a single data node in virtual scaled out architecture, then the system can maintain simple node-level deduplication logic, but storage efficiency deteriorates significantly in multi-node systems compared to single node architecture
Solution Approach 1:
The patent merges deduplication operations across multiple data nodes by introducing a global deduplication engine that coordinates file placement across the entire distributed system. Similar files are identified and consolidated onto the same node regardless of their physical location, achieving system-wide deduplication rather than isolated node-level deduplication.
Solution Approach 2:
The patent introduces a global deduplication engine as an intermediary component that sits between storage clients and data nodes. This intermediary receives file placement requests, determines optimal destination nodes based on similarity analysis, and redirects files to appropriate nodes, thereby enabling global deduplication without requiring complex modifications to individual node logic.
2Quantity of substance
If files are moved between data nodes to achieve global deduplication, then storage efficiency improves through better deduplication space savings, but data movement costs and network traffic increase
Solution Approach 1:
The patent implements partial action by selectively moving only those files that will contribute to deduplication savings, rather than relocating all files across the system. The global deduplication engine identifies specific file groups that benefit from consolidation and moves only those files, leaving other files in place, thereby reducing unnecessary data movement overhead.
Solution Approach 2:
The patent applies preliminary action by performing similarity analysis and determining optimal file placement destinations before actual file migration occurs. The global deduplication engine pre-calculates which files should be moved to which nodes based on content similarity, allowing the system to plan and execute efficient data movement that minimizes network traffic and maximizes deduplication benefits.
3Quantity of substance
If a global deduplication engine redistributes files across multiple nodes, then storage utilization improves through optimized file placement, but system complexity increases due to coordination overhead
Solution Approach 1:
The patent segments the global deduplication function into distinct modular components: a global deduplication engine for coordination, similarity analysis modules for identifying duplicate files, and file placement modules for executing migrations. This segmentation allows each component to perform its specific function independently, reducing overall system complexity while achieving global deduplication.
Solution Approach 2:
The patent implements self-service by enabling the global deduplication engine to automatically analyze file similarity, determine optimal placement locations, and execute file migrations without requiring manual intervention or complex external coordination. The system autonomously manages the entire deduplication process, reducing the operational complexity burden on system administrators.
Data Source
AI summary
A method, apparatus, and system for redistributing files in a multi-node storage system to improve global deduplication storage savings is disclosed. A plurality of file cluster candidates are generated for a plurality of files stored at a multi-node storage system comprising a plurality of data nodes. A similarity index is determined for each of the plurality of file cluster candidates based on similarity of the files comprised in the file cluster candidate. A ranked recipe list comprising a plurality of recipes is generated. Each recipe is associated with one of the plurality of file cluster candidates, comprises a destination data node for the associated file cluster candidate, and is associated with a deduplication space savings. At least some of the plurality of files are moved between the plurality of data nodes based on the recipes in the ranked recipe list to improve deduplication space savings in the multi-node storage system.