Deduplication-Aware Load Balancing in Distributed Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed storage systems that implement both load balancing and data deduplication fail to consider deduplicated data when making load balancing decisions, leading to suboptimal storage utilization.
Innovation Solution
The system incorporates a bloom filter and load balancing hints data structure to identify nodes with the most deduplicated data blocks, allowing for targeted rebalancing that minimizes new data allocation and accelerates the load balancing process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If existing load balancing mechanisms move data across nodes periodically to ensure storage utilization thresholds are met, then storage utilization is maintained, but storage utilization optimization is suboptimal because deduplicated data is not considered
Solution Approach 1:
The system implements feedback mechanisms by continuously monitoring deduplication metadata and bloom filters across nodes, using this information to dynamically adjust load balancing decisions. The load balancer receives feedback about which nodes have deduplicated blocks and uses this feedback to make informed data movement decisions that optimize storage utilization.
Solution Approach 2:
The invention changes the parameters considered in load balancing from simple storage capacity metrics to include deduplication-aware parameters such as the presence of deduplicated blocks (tracked via bloom filters) and deduplication metadata. This parameter change enables the system to identify target nodes that already have relevant deduplicated data, thereby optimizing storage utilization.
2Stability of the object's composition
If data is moved across nodes in traditional load balancing, then storage distribution is balanced, but data transfer time increases due to lack of awareness of existing deduplicated blocks
Solution Approach 1:
The system performs preliminary actions by pre-computing and maintaining bloom filters that indicate the presence of deduplicated blocks on each node before load balancing operations. This preliminary information preparation allows the load balancer to quickly identify target nodes that already have the required deduplicated data, avoiding unnecessary data transfers and reducing data transfer time.
Solution Approach 2:
Instead of copying entire data objects during load balancing, the system uses bloom filters (a compact data structure) to represent the presence of deduplicated blocks. This copying approach transfers minimal information (bloom filter bits) rather than actual data blocks, significantly reducing transfer time while maintaining the ability to make informed load balancing decisions.
3Reliability
If new data copies are created during load balancing, then data availability is maintained, but storage space is wasted due to duplication
Solution Approach 1:
The system applies local quality by making load balancing decisions specific to each target node's deduplication characteristics. By examining bloom filters locally at each node, the system identifies which nodes already have deduplicated blocks and directs data placement accordingly. This local analysis enables the system to place data only where necessary, maintaining data availability while minimizing redundant storage.
Solution Approach 2:
The system discards the traditional approach of creating full data copies during load balancing and recovers storage space by leveraging existing deduplicated blocks. When a target node already has the required deduplicated data (identified via bloom filters), the system discards the need to create new copies, thereby recovering storage space that would otherwise be wasted.
Data Source
AI summary
Techniques for enabling deduplication-aware load balancing in a distributed storage system are provided. In one set of embodiments, a node of the distributed storage system can receive an I/O (Input/Output) request pertaining to a data block of a storage object stored on a local storage component of the node. The node can further determine whether the I/O request requires insertion of a new entry into a deduplication hash table associated with the local storage component or deletion of an existing entry from the deduplication hash table. If the I/O request requires insertion of a new hash table entry, the node can add an identifier of the data block into a probabilistic data structure associated with the local storage component, where the probabilistic data structure is configured to maintain information regarding distinct data blocks that are likely present in the local storage component. Alternatively, if the I/O request requires deletion of an existing hash table entry, the node can remove the identifier of the data block from the probabilistic data structure.