Reference Table Sharding for Content-Addressable Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems face challenges in managing de-duplication and garbage collection efficiently, particularly with large objects and high object duplication rates, leading to storage resource wastage and network bandwidth issues.
Innovation Solution
A novel reference counting scheme for content-addressable storage clusters, where references are stored in a separate reference table with a content index, using a hash function for content identification and a write head derived from the object ID to distribute records across shards, enabling efficient deletion and garbage collection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If references are stored in a single reference table location, then lookup speed is improved, but hot spots occur causing performance degradation
Solution Approach 1:
The reference table is segmented into multiple shards distributed across different storage locations. Each shard stores a subset of references, preventing any single location from becoming a hot spot. This segmentation maintains lookup speed while improving performance stability by distributing the load across multiple nodes.
Solution Approach 2:
The system introduces a new dimension for reference storage by using a composite key structure that includes both content hash and object ID. This additional dimensional aspect allows references to be distributed across multiple shards while maintaining efficient lookup capabilities through the structured key organization.
2Quantity of substance
If de-duplication is implemented to save storage resources, then storage efficiency is improved, but reference counting complexity increases
Solution Approach 1:
The reference table structure serves multiple functions simultaneously: it tracks reference counts for de-duplication, enables efficient garbage collection, and provides lookup capabilities. This multi-functionality reduces the need for separate mechanisms, thereby managing complexity while maintaining storage efficiency.
Solution Approach 2:
The reference table acts as an intermediary structure between the content index and the actual stored objects. It mediates reference counting operations by maintaining a structured record of which objects reference which content, simplifying the overall system complexity while enabling effective de-duplication.
3Reliability
If garbage collection is performed as a separate batch process, then thoroughness is improved, but processing time increases
Solution Approach 1:
The reference table maintains up-to-date reference counts as objects are created, updated, or deleted. This preliminary tracking of reference information eliminates the need for extensive batch processing during garbage collection, as the system already has the necessary information to quickly identify unreferenced content.
Solution Approach 2:
The reference counting mechanism automatically updates itself through normal object operations. When objects are created or deleted, the reference table is automatically updated, making the garbage collection process self-servicing and reducing the time required for separate batch processing while maintaining thoroughness.
Data Source
AI summary
A process manages database storage. The process receives a first object comprising one or more content chunks. The first object is identified by a unique object ID and each content chunk has a unique offset within the first object. For each chunk, the process inserts a record into a reference table. The record includes a content hash and the object ID. The process stores each of the chunks in content storage. Later, the process obtains a request to delete a first chunk from storage. The first chunk has a corresponding first content hash. The process determines whether the reference table includes a reference record corresponding to the first content hash. When the reference table does not include any reference records corresponding to the first content hash, the process deletes the first chunk. When the reference table includes a corresponding reference record, the process does not delete the first chunk.