Cloud Storage Partition Rebalancing via Vector Clustering

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Cloud-based storage services face challenges in managing resource utilization across multiple partitions, leading to underutilization and overutilization of resources, requiring an efficient load balancing algorithm to redistribute user accounts without causing disruptions.

Innovation Solution

A load balancing algorithm that generates vectors for user accounts based on resource utilization parameters, sorts them into clusters, and maps accounts from heavily loaded partitions to less loaded ones, minimizing data movement and maintaining balanced resource consumption across partitions.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of manufacture

If a simple rebalancing algorithm that randomly hashes user accounts to various instances is used, then the implementation is trivial and easy to deploy, but a huge amount of data must be moved every time user accounts are rebalanced, which is time-consuming and causes disruptions

Engineering Contradiction:

Improveease of implementationVSAvoidrebalancing time

Core Design Contradiction:

Ease of manufactureVSLoss of time

Solution Approach 1:

The system performs preliminary actions by continuously monitoring resource utilization metrics and pre-identifying candidate user accounts for rebalancing before actual rebalancing is needed. This allows the system to prepare rebalancing plans in advance, reducing the actual execution time and minimizing disruptions when rebalancing occurs.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The rebalancing algorithm is made dynamic by continuously adapting to changing resource utilization conditions. The system adjusts rebalancing decisions based on real-time metrics, allowing flexible response to varying load conditions without requiring complete rebalancing cycles, thus reducing overall rebalancing time while maintaining effectiveness.

Inventive Principle:

Principle #15Dynamics

2Ease of manufacture

Engineering Contradiction:

Improveease of implementationVSAvoidservice continuity

Core Design Contradiction:

Ease of manufactureVSReliability

Solution Approach 1:

Instead of rebalancing all user accounts uniformly, the system applies partial action by selectively rebalancing only those user accounts that are identified as candidates based on resource utilization metrics. This reduces the scope of data movement while still achieving effective load balancing, thereby minimizing disruptions and maintaining service continuity.

Inventive Principle:

Principle #16Partial or excessive action

Solution Approach 2:

The system implements feedback mechanisms by continuously monitoring resource utilization metrics and using this information to guide rebalancing decisions. This closed-loop control ensures that rebalancing actions are taken only when necessary and are targeted appropriately, reducing unnecessary data movements and maintaining service stability while improving load distribution.

Inventive Principle:

Principle #23Feedback

3Quantity of substance

If user accounts are divided among multiple instances of the storage service, then the volume of data maintained by each instance is reduced and becomes more manageable, but utilization of storage resources by different user accounts varies significantly, leading to under-utilization and over-utilization of instances

Engineering Contradiction:

Improvedata volume per instanceVSAvoidresource utilization balance

Core Design Contradiction:

Quantity of substanceVSEase of operation

Solution Approach 1:

The system applies local quality by treating different user accounts and instances differently based on their specific resource utilization characteristics. Instead of uniform distribution, the algorithm identifies local imbalances in resource usage and targets rebalancing actions to specific instances and user accounts where they are most needed, optimizing overall resource utilization while maintaining manageable data volumes per instance.

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The system changes parameters by dynamically adjusting the distribution of user accounts based on monitored resource utilization metrics. The algorithm modifies assignment parameters in response to changing conditions, shifting user accounts from over-utilized instances to under-utilized ones, thereby balancing resource utilization across instances while keeping data volumes manageable.

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS11442632B2Rebalancing of user accounts among partitions of a storage service

Publication Date: 2022.09.13 APPLE INC

AI summary

This application relates to load balancing for a number of partitions of a network-based storage service. Each partition includes a number of server devices and/or network devices located in a data center and configured to provide access to storage resources hosted within the data center. User accounts are assigned to a particular partition such that requests related to a particular user account are routed to that partition. Periodically, a load balancing algorithm is executed to re-assign user accounts to different partitions to rebalance resource consumption across the different partitions. The load balancing algorithm can balance resource consumption for any number of resource types by generating a vector of resource utilization parameters for each user account, sorting the plurality of user accounts into clusters based on the vectors, and mapping at least some user accounts to different partitions.