Garbage Collection Scope Detection in Distributed Storage

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Determining the garbage collection scope in distributed storage systems is complex due to the need to identify unused storage chunks while ensuring referenced chunks are not mistakenly reclaimed, especially in systems using multi-version concurrency control and search trees.

Innovation Solution

A method is introduced that initializes a garbage collection front value based on the maximum sequence number of sealed storage chunks, uses this value to determine the garbage collection scope, retrieves metadata to identify unreferenced chunks, and reclaims storage capacity from these chunks, while advancing the front value during tree updates and sealing additional chunks.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of substance

If garbage collection is implemented at the chunk level in distributed storage systems, then storage space can be reclaimed, but determining the GC scope becomes complex and time-consuming

Engineering Contradiction:

Improvestorage space reclamationVSAvoidGC scope determination time

Core Design Contradiction:

Loss of substanceVSLoss of time

Solution Approach 1:

The system pre-assigns monotonically increasing sequence numbers to storage chunks in advance, establishing a natural ordering before garbage collection is needed. This preliminary numbering allows the GC process to efficiently determine scope by simply comparing sequence numbers, rather than analyzing complex reference relationships during the actual collection process

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The invention changes the approach from analyzing complex reference relationships to using a simple numerical parameter (sequence number) to define GC scope. By transitioning from a relationship-based determination method to a parameter-based method, the system achieves O(1) complexity for scope determination while maintaining storage space reclamation

Inventive Principle:

Principle #35Parameter changes

2Loss of substance

If all sealed storage chunks are considered for garbage collection, then more storage space can be reclaimed, but the risk of reclaiming referenced chunks increases

Engineering Contradiction:

Improvestorage space reclamationVSAvoiddata integrity

Core Design Contradiction:

Loss of substanceVSReliability

Solution Approach 1:

The system introduces sequence numbers as an intermediary parameter that mediates between the GC process and storage chunks. This intermediary allows the GC process to safely determine which chunks can be reclaimed by comparing sequence numbers without directly analyzing complex reference relationships, thus preventing referenced chunks from being mistakenly collected while maximizing space reclamation

Inventive Principle:

Principle #24Intermediary (Mediator)

3Ease of operation

If storage chunks are treated as immutable under MVCC, then concurrent access is simplified, but garbage collection can only occur at the chunk level

Engineering Contradiction:

Improveconcurrent accessVSAvoidgarbage collection process

Core Design Contradiction:

Ease of operationVSDevice complexity

Solution Approach 1:

The system segments the storage space into fixed-size chunks with monotonically increasing sequence numbers, creating a hierarchical structure. This segmentation allows the GC process to operate at the chunk level while maintaining simplicity, as each chunk is independently identifiable and manageable through its sequence number, avoiding the need for complex fine-grained collection mechanisms

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS10061697B2Garbage collection scope detection for distributed storage

Publication Date: 2018.08.28 EMC IP HLDG CO LLC

AI summary

Systems and methods for determining garbage collection (GC) scope in a distribute storage system using chunk-based storage. The systems and methods are compatible with multi-version concurrency control (MVCC) semantics.