Distributed Garbage Collection for Deduplicated Storage

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

In deduplicated storage systems, determining when data chunks are no longer referenced is complex and resource-intensive, particularly for large data sets, leading to storage fragmentation and wasted space due to dead or unreferenced data segments.

Innovation Solution

A distributed deduplication garbage collection process using strong hash functions and Bloom filters to identify live data segments, allowing for parallel processing and efficient deletion of unreferenced data segments by generating a garbage collection recipe that copies live data segments to new storage containers before deleting the old ones.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Quantity of substance

If traditional garbage collection methods are used to determine unreferenced data chunks, then storage space can be reclaimed, but the process becomes complex and resource-intensive for large datasets

Engineering Contradiction:

Improvestorage space reclamationVSAvoidgarbage collection process complexity

Core Design Contradiction:

Quantity of substanceVSDevice complexity

Solution Approach 1:

The patent introduces Bloom filters as intermediary data structures that enable efficient garbage collection by mediating between the metadata and the actual data chunks. These Bloom filters allow the system to determine whether data chunks are referenced without complex analysis, thus simplifying the garbage collection process while maintaining storage space reclamation effectiveness

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent replaces traditional mechanical garbage collection approaches with hash-based data structures (Bloom filters) and probabilistic algorithms. This substitution transforms the complex mechanical process of tracking references into efficient computational operations using hash functions and bit arrays, significantly reducing resource requirements for large datasets

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Productivity

If distributed processing is used to handle large datasets, then processing speed improves, but memory requirements increase

Engineering Contradiction:

Improveprocessing speedVSAvoidmemory requirements

Core Design Contradiction:

ProductivityVSQuantity of substance

Solution Approach 1:

The patent divides the distributed dataset into multiple segments, each with its own Bloom filter and hash table. This segmentation allows parallel processing across distributed nodes while keeping memory requirements localized to each node rather than requiring all nodes to hold the entire dataset in memory simultaneously

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent extracts only the essential reference information into Bloom filters, removing unnecessary data from memory. By storing only hashed representations of data chunk references rather than the actual reference relationships, the system achieves distributed processing capability with reduced memory requirements at each node

Inventive Principle:

Principle #2Taking out (Extraction)

Data Source

PatentUS10515009B1Method and system for reducing memory requirements during distributed garbage collection of deduplicated datasets

Publication Date: 2019.12.24 EMC IP HLDG CO LLC

AI summary

Embodiments relating to garbage collection for a deduplicated and compressed storage device are described. One embodiment provides for a networked system comprising a set of multiple computing devices, each computing device to provide one or more deduplication and replication services to one or more deduplicated storage containers and a central computing device connected to each computing device in the set of multiple computing devices over a local network, the central computing device to provide deduplication garbage collection to each computing device in the set of computing devices.