Decentralized Deduplication Using B+ Tree Hash Index
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional computer file systems inefficiently manage storage by storing data in fixed-size blocks, leading to redundant data blocks that consume unnecessary storage capacity, as they lack effective deduplication mechanisms to identify and merge identical blocks across multiple files.
Innovation Solution
A decentralized deduplication system where each host in a data center has its own deduplicating file system, using a hierarchical data structure to track write operations and asynchronously perform deduplication on storage blocks, allowing multiple hosts to cooperatively identify and merge identical data blocks without requiring access to all involved files, utilizing hash indices and write logs to manage and update metadata pointers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional file systems store data in fixed-size blocks without deduplication, then storage management is simple, but storage capacity is wasted due to redundant data blocks
Solution Approach 1:
The patent segments the deduplication system into distributed components, with each host maintaining its own deduplication capabilities independently. This segmentation allows the system to reduce storage capacity waste through deduplication without requiring a complex centralized system, as each host handles its own deduplication operations locally.
Solution Approach 2:
Each host in the distributed system performs self-service deduplication by autonomously identifying and merging duplicate data blocks within its own file system. This self-service approach eliminates the need for complex centralized coordination, reducing system complexity while improving storage capacity utilization through local deduplication operations.
2Quantity of substance
If a centralized deduplication system is implemented, then deduplication can be performed across all files, but system complexity and access coordination requirements increase
Solution Approach 1:
The patent divides the centralized deduplication approach into distributed segments, where each host independently performs deduplication on its own files. This segmentation eliminates the need for complex file access coordination across a centralized system, as each host operates autonomously while still achieving storage efficiency through local deduplication.
Solution Approach 2:
The patent introduces a metadata server as an intermediary that stores deduplication metadata without requiring hosts to coordinate file access. This intermediary approach allows hosts to perform deduplication independently while still achieving centralized storage efficiency, as the metadata server facilitates deduplication operations without imposing complex access coordination requirements.
3Quantity of substance
If deduplication is performed synchronously during write operations, then storage efficiency is improved, but performance penalties increase
Solution Approach 1:
The patent performs preliminary deduplication actions by pre-computing and storing metadata information during write operations, but the actual deduplication merging is deferred. This preliminary action approach allows write operations to complete quickly without performance penalties, while storage efficiency is still improved through subsequent async deduplication operations that occur in the background.
Solution Approach 2:
The patent implements periodic async deduplication operations that periodically merge duplicate blocks identified during write operations. This periodic action approach separates the write operation performance from the deduplication efficiency, allowing write operations to complete without performance penalties while storage efficiency is improved through periodic background deduplication tasks.
4Quantity of substance
If all hosts must have access to all involved files for deduplication, then complete deduplication can be achieved, but system scalability is limited
Solution Approach 1:
The patent introduces a metadata server as an intermediary that enables complete deduplication across all hosts without requiring direct file access between hosts. This intermediary approach maintains deduplication completeness by centralizing metadata management while improving system scalability, as new hosts can join the system without requiring access to files on existing hosts.
Data Source
AI summary
Decentralized deduplication operations in a computer system employ a hash index that is a variant of a B+ tree to support both efficient sequential updates as well as efficient random updates. Sequential update is selected when deduplication is infrequently performed, such as on the order of days, and random update is selected when deduplication is performed more frequently, such as on the order of seconds. More frequent deduplication may be beneficial during periods when large amounts of temporary duplicate data are created, and the system may not have enough storage space to accommodate the temporary spike in demand.