Deduplicated Data Store Selection via Bidding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current backup and recovery systems in computer networks face inefficiencies in data deduplication and storage distribution across multiple nodes, leading to complexities in managing and retrieving data effectively.
Innovation Solution
A distributed data storage system where 'teamed objects' are stored across multiple independent nodes, utilizing a bidding process to route similar data chunks to the same node and employing deduplicated data storage with a sparse hash index to enhance deduplication efficiency and load balancing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is distributed across multiple nodes for backup storage, then reliability and capacity are improved, but data management complexity and retrieval efficiency deteriorate
Solution Approach 1:
The patent divides backup data into deduplicated data chunks and distributes them across multiple team members (nodes). Each node stores a portion of the deduplicated data, identified by unique hashes. This segmentation improves reliability through distribution while managing complexity via automated hash-based routing and bidding processes that handle node selection and data placement.
Solution Approach 2:
The patent introduces a bidding process as an intermediary mechanism between data placement requests and node selection. Team members bid to receive specific deduplicated data based on their capacity and suitability, allowing the system to optimize data distribution automatically. This intermediary layer simplifies management complexity by replacing manual allocation with an automated competitive selection process.
2Quantity of substance
If deduplication is implemented across distributed nodes, then storage efficiency is improved, but the complexity of tracking and managing deduplicated data increases
Solution Approach 1:
The patent uses hash copies as identifiers for deduplicated data chunks. Instead of tracking entire data objects, the system creates and distributes hash copies that uniquely identify each deduplicated chunk. These hashes are communicated to team members during the bidding process, enabling efficient tracking and retrieval without managing complex metadata about the original data structures.
Solution Approach 2:
The patent transforms data identification from object-level tracking to hash-value-based tracking. By changing the parameter used to identify data (from full object metadata to compact hash values), the system achieves better storage efficiency while reducing tracking complexity. The hash values serve as simplified parameters that enable quick matching and routing without requiring complex data structure management.
3Productivity
If similar data chunks are routed to the same node, then deduplication ratio is improved, but load balancing across nodes deteriorates
Solution Approach 1:
The patent implements a dynamic bidding process where team members can bid for deduplicated data based on their current capacity and load conditions. This dynamic approach allows the system to adaptively balance loads: when one node becomes overloaded, other nodes can bid to receive data chunks, ensuring both high deduplication ratios (by routing similar data to the same node when possible) and good load balancing (by allowing flexible reassignment based on current node states).
Data Source
AI summary
A technique includes communicating signatures of samples of first data associated with an object to at least some stores of a plurality of stores on which the object is distributed, and in response to the communication, receiving responses indicating numbers of the samples stored on the respective at least some of the plurality of stores and regulating on which store of the plurality of stores deduplicated data associated with the first data is stored based at least in part on the numbers and a pattern of data storage on the stores.