Distributed Deduplication Storage Messaging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deduplication storage systems face challenges in being easily expandable without compromising storage savings, as they either struggle with maintaining high deduplication performance or sacrificing storage efficiency during expansion.
Innovation Solution
A deduplication storage system that uses queries to identify storage devices with duplicate data, employing fingerprint generation and communication methods such as broadcasted or directed queries to optimize duplicate data recognition across multiple devices, allowing for seamless expansion without requiring downtime.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Volume of stationary object
If deduplication storage systems are expanded to multiple storage devices, then storage capacity and scalability are improved, but system complexity and difficulty in managing duplicate data across devices increase
Solution Approach 1:
The system divides the deduplication management into segments: storage array controllers manage individual storage arrays, each with its own fingerprint database. The messaging system enables coordinated communication between controllers. This segmentation allows each controller to independently manage its storage devices while maintaining system-wide deduplication through inter-controller messaging.
Solution Approach 2:
A messaging system acts as an intermediary between storage array controllers, enabling them to share deduplication information and coordinate duplicate data management. The messaging infrastructure facilitates information exchange about fingerprints and duplicate data locations without requiring complex direct integration between all controllers, thus reducing overall system complexity.
2Measurement precision
If broadcasted queries are sent to all storage devices to identify duplicate data, then duplicate data recognition is improved, but bandwidth consumption increases
Solution Approach 1:
The system implements local quality by having each storage array controller maintain its own fingerprint database for its managed storage devices. Controllers first query their local fingerprints before broadcasting to other controllers. This local optimization reduces the need for widespread broadcasting, as many duplicate data identifications can be made locally without consuming additional bandwidth.
Solution Approach 2:
Instead of broadcasting queries to all storage devices in the entire system, the messaging system enables targeted queries between controllers only when necessary. Controllers send messages to specific peer controllers based on data distribution patterns, performing partial action rather than exhaustive broadcasting, thus reducing overall bandwidth consumption while maintaining effective duplicate recognition.
3Speed
If directed queries are sent over dedicated communication channels between storage devices, then communication efficiency is improved, but device complexity and infrastructure requirements increase
Solution Approach 1:
The messaging system provides a universal communication infrastructure that serves multiple functions: it enables directed queries between controllers, facilitates fingerprint database synchronization, and supports coordinate management for duplicate data. This single multi-functional messaging layer replaces the need for separate dedicated communication channels for each function, reducing infrastructure complexity while maintaining communication efficiency.
Data Source
AI summary
Receiving, at a storage array controller, a write instruction that includes a logical address and write data. Using the storage array controller to generate a fingerprint from the write data and sending, from the storage array controller to one or more recipient storage devices, a query that includes the fingerprint, wherein the query asks the recipient storage devices if the fingerprint is stored on any of the recipient storage devices.