Distributed Storage Deduplication via Hash Entry Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional storage systems face challenges in efficiently integrating data deduplication and cloning, leading to lower space efficiencies and scalability issues due to centralized architectures that do not effectively share data blocks across multiple logical entities.
Innovation Solution
A distributed storage system that uses hash entry objects and logical pointer entry objects to manage deduplicated data, where each physical data block is associated with a hash entry containing a computed value, address, and reference count, and each logical data block is associated with a pointer to the hash entry, allowing for global data block sharing and efficient deduplication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If centralized deduplication architecture is used, then implementation simplicity is improved, but storage space efficiency deteriorates due to inability to share data blocks globally
Solution Approach 1:
The system segments the storage architecture into distributed storage nodes that independently manage local data blocks, while a centralized index service coordinates global deduplication. This allows each node to operate autonomously (maintaining simplicity) while participating in global deduplication (improving space efficiency).
Solution Approach 2:
The patent introduces a new dimension of organization by separating data storage (physical dimension at node level) from data indexing (logical dimension at system level). The index service operates in the logical dimension to track all data blocks across the distributed system, enabling global deduplication without complicating individual node operations.
2Adaptability or versatility
If localized deduplication within individual devices or groups is performed, then device autonomy is improved, but overall space efficiency deteriorates
Solution Approach 1:
The system merges localized deduplication capabilities at each storage node with centralized index coordination. Each node maintains autonomous deduplication operations locally, while the centralized index service merges information from all nodes to enable cross-node deduplication, achieving both device autonomy and global space efficiency.
Solution Approach 2:
The index service performs multiple functions: it tracks data blocks for global deduplication, manages reference counts across the distributed system, and coordinates cloning operations. This universal service enables autonomous nodes to participate in system-wide deduplication without sacrificing their independence.
3Quantity of substance
If data blocks are shared globally across multiple logical entities, then storage space efficiency is improved, but system complexity deteriorates
Solution Approach 1:
The centralized index service acts as an intermediary between distributed storage nodes and logical entities. It manages the complexity of tracking and coordinating data block sharing across the distributed system, while nodes simply store and serve data blocks. This intermediary absorbs the system complexity, allowing nodes to remain simple.
Solution Approach 2:
The system implements feedback mechanisms where the index service continuously tracks reference counts of shared data blocks and coordinates updates across nodes. When a logical entity accesses or modifies data, the index service receives feedback and updates the global state, ensuring consistency without requiring complex peer-to-peer coordination between nodes.
4Ease of operation
If conventional centralized architecture is used, then management simplicity is improved, but scalability deteriorates due to single data path bottleneck
Solution Approach 1:
The system segments the centralized management function into a lightweight index service that handles only metadata operations, while data operations are distributed across multiple nodes. This segmentation removes the bottleneck from the management path, allowing the index service to remain simple while the data paths scale horizontally across nodes.
Solution Approach 2:
The patent separates management operations (index updates, deduplication coordination) from data operations (read/write access). Management occurs in the metadata dimension through the index service, while data operations occur in the storage dimension across distributed nodes. This dimensional separation allows independent scaling of data capacity without managing complexity.
Data Source
AI summary
A method and system are disclosed for storing client data objects in a deduplicated storage system. Deduplicated data may be stored in a plurality of physical data blocks. A content map layer can provide a mapping between the physical data blocks to logical map objects associated with the client data objects. The deduplicated data may be mapped to logical data blocks that comprise the client data objects.