Clustered File Deduplication Across Storage Tiers Using Similarity Groups
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deduplication processes in clustered networks with multiple storage tiers are inefficient, as they fail to optimize data deduplication across different storage tiers with varying cost and performance characteristics.
Innovation Solution
A multi-tier deduplication manager optimally distributes data across multiple storage tiers by partitioning the data space into similarity groups and assigning these groups to dedicated deduplication services within each tier, allowing for local deduplication checks without remote procedure calls, thus maximizing global deduplication efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If data is deduplicated across multiple storage tiers using traditional processes, then storage overhead is reduced, but deduplication efficiency deteriorates due to remote procedure call overhead and lack of optimization for tier-specific characteristics
Solution Approach 1:
The patent segments the deduplication process by creating tier-specific deduplication services for each storage tier. Each service handles deduplication locally within its tier using similarity groups, eliminating the need for centralized remote procedure calls and optimizing for tier-specific characteristics.
Solution Approach 2:
The patent introduces a new dimension of organization by implementing similarity groups that partition data space across tiers. This creates a hierarchical structure where deduplication operates both within tiers and across tiers simultaneously, improving efficiency without sacrificing storage optimization.
2Speed
If deduplication processes are optimized for one storage tier, then performance on that tier improves, but global deduplication efficiency deteriorates due to lack of cross-tier optimization
Solution Approach 1:
The patent creates tier-specific deduplication services that are universal in their approach but specialized in implementation. Each service follows the same similarity group methodology, allowing consistent cross-tier optimization while adapting to individual tier characteristics for maximum performance.
3Productivity
If centralized deduplication management is used across all tiers, then global deduplication is achieved, but system complexity increases due to remote procedure calls and centralized coordination overhead
Solution Approach 1:
The patent merges the deduplication logic across all tiers into a unified similarity group framework. While services are distributed, they operate under consistent rules and data structures, achieving global deduplication without the complexity of centralized coordination or remote procedure calls.
Solution Approach 2:
Each storage tier operates its own deduplication service autonomously using similarity groups. The system achieves global deduplication through self-service mechanisms where each tier independently manages its data while contributing to overall deduplication efficiency without requiring centralized control.
Data Source
AI summary
Embodiments are described for a system and method facilitating deduplication in a multi-tier storage system in which a file can have different portions written to different tiers. A process partition the data space of each tier to a number of similarity groups and distributes the similarity groups across file system services in a cluster. The distribution is done in such a way that for a given similarity group ID, the same file system service owns the similarity groups of every tier. This allows for efficient checks for deduplication as it can be done local to a node rather than requiring remote procedure calls.