Uniqueness Index File Selection for Backup Load Balancing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In deduplication backup systems, efficiently moving files between nodes in a cluster-based storage system is challenging, as it often results in uneven data distribution and loss of deduplication advantages, making it difficult to maintain efficient deduplication processes.
Innovation Solution
The Mostly Unique File Selection (MUFS) process, which uses a uniqueness index (u-index) and dynamic memory allocation to select and migrate files with high similarity, ensuring that the space freed up is proportional to the data moved, thereby optimizing space savings and maintaining deduplication effectiveness.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If files are moved between nodes in a deduplication backup system, then load balancing is improved, but deduplication effectiveness is lost
Solution Approach 1:
The patent introduces an intermediary mechanism (the file selection algorithm that calculates uniqueness indices) to mediate between the need for load balancing and the need to preserve deduplication effectiveness. This intermediary selects which files to move based on their uniqueness characteristics, ensuring that moves are made intelligently rather than arbitrarily.
Solution Approach 2:
The patent changes the parameter selection criteria for file migration by introducing a uniqueness index calculation. Instead of moving files based on simple load balancing metrics, the system now considers the uniqueness index of files, which reflects their deduplication potential. This parameter change allows the system to balance load while preserving deduplication effectiveness.
2Ease of manufacture
If random file selection is used for migration, then implementation simplicity is improved, but space freeing efficiency deteriorates
Solution Approach 1:
The patent changes the selection parameter from random to uniqueness-index-based. By calculating and using the uniqueness index as the selection criterion, the system transforms the file selection process from a simple random operation to a targeted selection process that optimizes space freeing efficiency.
Solution Approach 2:
The patent performs preliminary action by calculating the uniqueness index of files before selecting them for migration. This pre-calculation allows the system to identify which files will be most effective to move for space recovery, rather than relying on random selection.
3Speed
If traditional file selection methods are used, then processing speed is improved, but deduplication optimization deteriorates
Solution Approach 1:
The patent introduces a new parameter (uniqueness index) that combines both speed and optimization considerations. By calculating this index efficiently and using it as the selection criterion, the system achieves both fast processing and optimal deduplication results.
Solution Approach 2:
The patent replaces traditional mechanical file selection methods (random or round-robin selection) with a computational approach based on uniqueness index calculation. This substitution allows for more intelligent decision-making while maintaining processing efficiency.
Data Source
AI summary
Embodiments for a mostly unique file selection process for a deduplication backup system are described. The process assigns tags to files. A tag serves as a hint about the similarity of files in a deduplication file system. It is expected that files from the same client machine will be assigned the same tag. The tag is the smallest unit of migration and serves as a hint of the similarity of the files. The MUFS process measures the uniqueness using a u-index that is a function of the total unique size of a tag relative to the total size of the tag. A load balancer then selects the most unique tags for migration to free the maximum space. It uses the u-index to measure the uniqueness percentage of a tag, so that tags with the highest u-index are selected for migration to free up maximum space on the source node.