Content-Aware Load Balancing for Backup Storage Systems

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

In disk-based backup systems, the assignment of clients to storage appliances becomes complex due to deduplication, leading to issues with capacity and throughput, as clients may experience overload or inefficiencies when moved between appliances, necessitating automated load balancing to optimize deduplication benefits.

Innovation Solution

A content-aware load balancing system that automatically assigns storage units to clients based on capacity, throughput, and deduplication rates, using a cost function to minimize penalties related to overload and data movement, and dynamically reallocates data for improved performance and deduplication efficiency.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If clients are assigned to storage appliances based on simple load balancing, then throughput is improved, but deduplication efficiency deteriorates

Engineering Contradiction:

ImprovethroughputVSAvoiddeduplication efficiency

Core Design Contradiction:

ProductivityVSLoss of energy

Solution Approach 1:

The system changes the parameter of client assignment from simple load-based metrics to content-aware metrics by comparing data fingerprints. This allows the system to maintain high throughput while preserving deduplication efficiency by assigning clients to storage appliances based on data similarity rather than just current load levels.

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The system implements feedback by continuously monitoring data fingerprints and comparing them across storage appliances. This feedback mechanism enables dynamic reassignment of clients to optimize both throughput and deduplication by identifying which storage appliances already contain similar data.

Inventive Principle:

Principle #23Feedback

2Reliability

If clients are moved between storage appliances to balance load, then overload is reduced, but data movement overhead increases

Engineering Contradiction:

Improveload balancingVSAvoiddata movement overhead

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system performs preliminary action by comparing data fingerprints before moving clients between storage appliances. This allows the system to identify assignment opportunities that will actually benefit from movement, reducing unnecessary data transfers and minimizing overhead while still achieving load balancing.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system changes the decision parameter for client movement from purely load-based metrics to a combination of load metrics and data similarity metrics. This reduces unnecessary movements by only transferring clients when both load balancing and deduplication benefits are present.

Inventive Principle:

Principle #35Parameter changes

3Ease of operation

If manual assignment of clients to storage appliances is used, then control over data placement is improved, but system complexity increases

Engineering Contradiction:

Improvecontrol over data placementVSAvoidsystem complexity

Core Design Contradiction:

Ease of operationVSDevice complexity

Solution Approach 1:

The system implements self-service by automatically comparing data fingerprints and making assignment decisions without manual intervention. The content-aware load balancing algorithm autonomously optimizes client-to-storage-appliance assignments based on current system state, eliminating the need for manual configuration while maintaining control.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The system uses feedback from fingerprint comparison results to automatically adjust assignments. This closed-loop control enables the system to maintain optimal data placement automatically, providing control equivalent to manual assignment but with reduced complexity through automation.

Inventive Principle:

Principle #23Feedback

4Quantity of substance

If storage capacity is increased to handle all clients, then capacity requirements are met, but storage utilization efficiency decreases

Engineering Contradiction:

Improvestorage capacityVSAvoidstorage utilization efficiency

Core Design Contradiction:

Quantity of substanceVSProductivity

Solution Approach 1:

The system changes the assignment strategy from static capacity-based allocation to dynamic content-based allocation. By comparing data fingerprints and identifying similarities, the system can consolidate similar data onto the same storage appliances, improving utilization efficiency without requiring additional capacity.

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS9875163B1Method for replicating data in a backup storage system using a cost function

Publication Date: 2018.01.23 EMC IP HLDG CO LLC

AI summary

An operating state of each of a plurality of storage units of a storage system is periodically monitored, including a storage capacity, a throughput, and overlap of clients associated with the storage units. In response to a request to redistribute data from a first of the storage units to another storage unit, a cost factor for each of remaining storage units to relocate the data of the first storage unit to each of the remaining storage units is determined. A cost factor of each of the remaining storage units is determined based on at least one of the storage capacity, the throughput, or the overlap of clients of the storage unit. A second of the storage units having a lowest cost factor amongst the remaining storage units is selected. At least a portion of the data of the first storage unit is migrated to the second storage unit.