Distributed Object Storage Partitioning for Cluster Load Balance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing object storage services face local cluster overload issues, where the load of local nodes exceeds the processing capability, affecting service availability.
Innovation Solution
A distributed object storage method that splits target objects into key-value pair data, determines target partitions using hash processing, and caches data onto multiple data storage nodes based on a partition view, ensuring load balance and redundancy through replica data distribution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If object storage services are provided using existing methods, then data storage requirements are met, but local cluster overload occurs where node load exceeds processing capability
Solution Approach 1:
The patent segments the object storage system into multiple partitions distributed across different nodes. Each partition independently stores a portion of the object data, preventing any single node from bearing excessive load. This segmentation allows the system to scale horizontally while maintaining service availability even when individual nodes are overloaded.
2Productivity
If data is stored on multiple data storage nodes, then load balance is improved, but system complexity increases due to partition management
Solution Approach 1:
The patent introduces a partition view as an intermediary data structure that manages the mapping between objects and partitions. This partition view abstracts the complexity of distributed partition management, providing a unified interface for storing and retrieving data across multiple nodes without requiring complex coordination logic in the storage operations themselves.
3Reliability
If replica data is distributed across different nodes, then disaster tolerance is enhanced, but storage space requirements increase
Solution Approach 1:
The patent applies local quality by distributing replica data across different physical nodes based on the partition view. Each node stores replicas of specific partitions, creating localized redundancy that enhances disaster tolerance. This approach optimizes storage space by ensuring that replicas are strategically placed rather than uniformly distributed, reducing unnecessary duplication while maintaining reliability.
Data Source
AI summary
A distributed object storage method, a device and a medium are provided, and the method includes: splitting a target object to be stored into a quantity of key-value pair data; determining target partitions corresponding respectively to the quantity of key-value pair data from a plurality of partitions of a preset distributed object storage system; determining, based on a preset partition view, data storage nodes that currently hold the target partitions, to obtain data storage nodes corresponding respectively to the quantity of key-value pair data; and caching the quantity of key-value pair data into the data storage nodes corresponding respectively to the quantity of key-value pair data.