Deduplicated Data Storage via Node Bidding Mechanism
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional backup and recovery systems face inefficiencies in data deduplication and storage distribution across multiple nodes, leading to complexities in managing and retrieving data, especially in distributed environments.
Innovation Solution
A system where data is stored as 'teamed objects' across a cluster of nodes, utilizing a bidding process to route similar data chunks to the most suitable node for deduplication, and employing a distributed master manifest to track data storage, allowing for efficient deduplication and redundancy while minimizing dependencies and single points of failure.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is distributed across multiple nodes for redundancy and efficiency, then storage efficiency and deduplication ratios improve, but system complexity and difficulty of managing data retrieval increase
Solution Approach 1:
The patent introduces a bidding coordinator as an intermediary component that manages the complex interactions between multiple storage nodes. This coordinator receives bid responses from nodes, selects the winning node for data storage, and manages the distributed data placement logic, thereby isolating the complexity from the overall system while maintaining high storage efficiency through distributed deduplication across multiple nodes.
2Ease of operation
If a centralized system manages all data storage and retrieval, then data management is simplified, but single points of failure and bottlenecks increase
Solution Approach 1:
The patent segments the centralized data management function into distributed components across multiple storage nodes. Each node independently participates in the bidding process and can store data chunks, eliminating the single point of failure that would exist in a fully centralized system. The segmentation allows the system to maintain operational simplicity through standardized node interfaces while achieving reliability through distribution.
3Quantity of substance
If deduplication is performed across the entire dataset, then storage efficiency improves, but processing time and computational resources increase
Solution Approach 1:
The patent implements partial deduplication action by having storage nodes bid for specific data chunks rather than performing exhaustive deduplication across the entire dataset simultaneously. Nodes submit bids for chunks they can efficiently store, and the system processes bids in a controlled manner, achieving significant deduplication ratios without the prohibitive processing time that would result from analyzing the complete dataset at once.
Data Source
AI summary
A technique includes communicating a plurality of hashes associated with chunks of an object to at least some stores of a plurality of stores on which the object is distributed; and in response to the communication, receiving responses indicating a distribution of the associated chunks. The technique includes selecting one of the stores based at least in part on the responses and communicating deduplicated data associated with the object to the selected store.