Multi-Cluster Storage Sharding for Scalable Capacity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current virtual computing systems face limitations in scaling storage capacity without increasing complexity, as traditional methods like adding disks or nodes are constrained by space, cost, and power consumption, and managing multiple namespaces becomes complex.
Innovation Solution
A multi-cluster storage system that monitors usage across clusters, alerts users when capacity thresholds are exceeded, migrates data, and removes clusters, allowing for scalable storage expansion by leveraging free capacity in other clusters, thereby federating clusters to minimize latency and prevent provider lock-in.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional storage expansion methods (adding disks or nodes) are used, then storage capacity is increased, but device complexity and constraints (space, cost, power consumption) increase
Solution Approach 1:
The storage system is segmented into multiple independent clusters, each managing its own storage resources. The object store is divided into multiple namespaces that can be distributed across clusters. This segmentation allows storage capacity to scale across clusters without increasing the complexity of a single monolithic system, as each cluster operates independently while contributing to the overall storage capacity.
Solution Approach 2:
The patent transitions from single-cluster storage to multi-cluster storage architecture, adding a new dimension of distribution across clusters. Instead of expanding storage within a single cluster boundary, the system distributes storage across multiple clusters, thereby increasing capacity without proportionally increasing the complexity of individual cluster components.
2Adaptability or versatility
If multiple namespaces are managed in traditional systems, then storage flexibility is improved, but management complexity increases
Solution Approach 1:
The object store is designed with universal namespace management that can operate across multiple clusters simultaneously. A single object store instance can manage multiple namespaces distributed across different clusters, providing storage flexibility without requiring separate management systems for each namespace. The system provides unified namespace operations (create, read, update, delete) that work consistently across the distributed multi-cluster environment.
3Speed
If data is stored in a single location, then access speed is simplified, but latency increases when serving distributed users
Solution Approach 1:
The patent implements local quality by serving data from the closest available cluster to the user, rather than from a single centralized location. Different regions of the distributed storage system have different characteristics (locality to users), and the system selects the optimal location for serving each request based on proximity. This reduces latency for distributed users while maintaining efficient data access through localized serving.
4Reliability
If storage resources are concentrated in one cluster, then resource management is simplified, but fault tolerance decreases
Solution Approach 1:
Storage resources are segmented and distributed across multiple independent clusters rather than concentrated in one. This segmentation provides fault tolerance because failures in one cluster do not affect others. The system manages this distributed resource allocation through namespace-based organization, where each namespace can be replicated or distributed across clusters, providing redundancy without requiring complex inter-cluster resource management protocols.
Data Source
AI summary
An illustrative embodiment disclosed herein is an apparatus including a processor having programmed instructions to maintain an object store including a primary cluster having one or more compute resources and one or more first storage resources, identify a secondary cluster having one or more second storage resources, select the secondary cluster to be added to the object store, allocate an available portion of the one or more second storage resources to the object store, and shard an object across the one or more second storage resources and the available portion of the one or more second storage resources.