Data Redirector for Sharded Deduplication Pools
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems that employ distributed blockpools require significant re-architecture and incur high locking latency costs, limiting the scalability and efficiency of data distribution across multiple nodes.
Innovation Solution
The implementation of a data redirector that facilitates BLOB-level or blocklet-level redirection, allowing for the distribution of data across multiple deduplication pools with minimal modification to existing systems, thereby achieving linear performance scaling and reduced redundant data elimination loss.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is distributed across multiple nodes using existing distributed blockpool approaches, then storage capacity and parallelism are improved, but locking latency increases and system complexity increases
Solution Approach 1:
The patent segments the distributed blockpool into multiple independent sharded blockpools, where each sharded blockpool operates autonomously without requiring cluster-level locks. This segmentation eliminates the need for global locking mechanisms while maintaining distributed storage capacity across multiple nodes.
Solution Approach 2:
The patent introduces a data redirector as an intermediary component that sits between clients and sharded blockpools. The redirector handles data routing and coordination, allowing sharded blockpools to operate independently without requiring complex inter-node locking protocols, thus reducing locking latency while maintaining system coherence.
2Productivity
If data is distributed across multiple nodes, then storage scalability is improved, but device complexity increases due to re-architecture requirements
Solution Approach 1:
The patent creates sharded blockpools that are functionally equivalent to traditional blockpools but with added sharding capability. Each sharded blockpool can independently perform deduplication, compression, and data storage operations, allowing the system to scale by simply adding more identical units without requiring fundamental architectural changes to existing components.
Solution Approach 2:
The data redirector serves as a universal intermediary that handles all client requests to multiple sharded blockpools. It abstracts the complexity of data distribution and routing from individual blockpool implementations, allowing each sharded blockpool to remain relatively simple while the system as a whole achieves high scalability.
3Reliability
If fine-grained locking is implemented across nodes, then data consistency is improved, but locking latency dominates parallelism gains
Solution Approach 1:
The patent segments data into distinct shards distributed across different blockpools, where each shard can be independently accessed and modified. This segmentation allows multiple parallel operations on different shards without requiring locks on the entire dataset, maintaining data consistency within each shard while eliminating the need for global locking that would serialize operations.
Solution Approach 2:
The data redirector acts as a mediator that coordinates access to sharded data, using hash-based routing to direct operations to specific shards. This intermediary approach ensures data consistency by routing related operations to the same shard while allowing completely independent parallel operations on different shards, avoiding the need for coarse-grained locks that would dominate parallelism gains.
Data Source
AI summary
Embodiments redirect data from a source to a plurality of deduplication blockpools. In one embodiment, operations include accessing a binary large object (BLOB) that includes a plurality of blocklets, a blocklet having a hash value; upon determining that the plurality of blocklets includes less than a threshold number of blocklets: selecting, according to a first rule set, a target blockpool from among the plurality of deduplication blockpools; upon determining that the plurality of blocklets includes at least the threshold number of blocklets: selecting, according to a second, different rule set, a target blockpool. The second rule set includes a BalanceQuery rule that computes a fitness value for a member of the plurality of deduplication blockpools, and selects a target blockpool based, at least in part, on the fitness value. Embodiments provide the BLOB to the target blockpool.