Clustered Storage Deduplication via Placement Tags

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Deduplicated backup data management in clustered storage systems faces inefficiencies due to uneven distribution of data across nodes, leading to deduplication inefficiencies and potential storage imbalances, which can result in backup failures and performance issues.

Innovation Solution

The system employs granular data source identification to intelligently distribute backup files across storage nodes within a cluster, using placement tags to group related data and optimize deduplication efficiency, while also rebalancing storage space through intelligent incremental backup handling and concurrent data movement techniques.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If data is evenly distributed across cluster nodes, then storage capacity and resource utilization are balanced, but deduplication efficiency is reduced

Engineering Contradiction:

Improvestorage capacity utilizationVSAvoiddeduplication efficiency

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The system applies different distribution strategies to different data types: related data (same data source) is kept localized on the same node to maintain deduplication efficiency, while unrelated data is evenly distributed across nodes to balance storage capacity. This local quality differentiation resolves the contradiction between balanced storage utilization and deduplication efficiency.

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The system segments data into related data (from the same data source) and unrelated data, applying different placement strategies to each segment. Related data is placed on the same node to preserve deduplication relationships, while unrelated data is distributed evenly for load balancing, thus resolving the contradiction through selective segmentation.

Inventive Principle:

Principle #1Segmentation

2Productivity

If related backup data is stored on different nodes, then storage load is balanced, but deduplication efficiency is lost

Engineering Contradiction:

Improvestorage load distributionVSAvoiddeduplication efficiency

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The system makes related backup data 'special' by keeping it localized on the same node through placement tags, while allowing unrelated data to be distributed for load balancing. This local quality approach preserves deduplication efficiency for related data while achieving overall storage balance.

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

Instead of distributing all data evenly (the conventional approach), the system inverts the strategy by concentrating related data on the same node. This inversion preserves deduplication relationships while the overall system still achieves load balance through distributed unrelated data.

Inventive Principle:

Principle #13The other way round (Inversion)

3Reliability

If incremental backups are performed during node rebalancing, then continuous backup availability is maintained, but performance overhead increases

Engineering Contradiction:

Improvebackup availabilityVSAvoidperformance overhead

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The system performs preliminary identification of related data using placement tags before rebalancing occurs. This preliminary action allows the system to pre-identify which data should remain on the original node, enabling incremental backups to proceed without scanning or analyzing data relationships during the actual backup operation, thus reducing performance overhead while maintaining availability.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The placement tag mechanism enables data to 'self-identify' its relationships with other backup data. During incremental backups, the system leverages this self-service identification without requiring complex analysis, reducing computational overhead while ensuring continuous backup availability during rebalancing operations.

Inventive Principle:

Principle #25Self-service

Data Source

PatentUS11520665B2Optimizing incremental backup for clients in a dedupe cluster to provide faster backup windows with high dedupe and minimal overhead

Publication Date: 2022.12.06 EMC IP HLDG CO LLC

AI summary

An intelligent method of handling incremental backups concurrent with load balancing movement. The file system uses placement tags, incremental backup requests and capacity balancing data movement to make intelligent decision to avoid affecting any backup windows for clients or backup apps. The file system tracks capacity balancing file movements inside the cluster. When switching locations of files in a cluster from one node to another, it is performed as an atomic change of switching inode attributes by the file system after the contents of the file have been copied over to the new node. During the file movement for capacity balancing, the file system handles requests for full backups differently than requests for incremental backups. The file system continues to handle virtual systhesis and fastcopy requests on the node that hosts the previous backup to ensure that the incremental backup succeeds with the expected smaller backup window from the client.