Adaptive Consistent Hashing for Dynamic Data Shard Allocation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for large-scale data storage systems fail to address the issue of data hotspots and storage imbalance, and the storage system's overall throughput and read/write performance are affected by fixed shard numbers, leading to storage imbalance and data hotspots.
Innovation Solution
A method and apparatus that dynamically allocates storage units based on the data volume of target data using an adaptive consistent hashing processing module, synchronizing data shard information to a configuration center, and writing data into corresponding storage engines to support query processing, thereby avoiding data hotspots and improving system performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If the number of shards is fixed based on the number of available storage units, then the storage system can be简单易实施, but it causes data hotspots and storage imbalance
Solution Approach 1:
The patent applies dynamics by making the number of shards variable rather than fixed. The system dynamically adjusts the number of shards based on the actual data volume and characteristics. Specifically, the patent divides data into different feature groups and determines the number of shards for each group based on its data volume, allowing the storage system to adapt to changing data distribution patterns and avoid data hotspots while maintaining implementation feasibility.
2Device complexity
If the number of shards is fixed, then the system structure is simple, but it affects overall throughput and read/write performance
Solution Approach 1:
The patent applies segmentation by dividing the storage system into multiple feature groups based on data characteristics. Each feature group has its own shard configuration optimized for its specific data volume and access patterns. This segmentation allows different parts of the system to operate at optimal performance levels simultaneously, improving overall throughput while keeping each segment's structure relatively simple and manageable.
3Ease of operation
If data is stored in a fixed number of shards, then storage allocation is straightforward, but it leads to large data sets being stored in a single unit causing data hotspots
Solution Approach 1:
The patent applies parameter changes by dynamically adjusting the number of shards as a key parameter based on data volume. For feature groups with larger data volumes, the system increases the number of shards to distribute data more evenly across storage units. This parameter adjustment is performed automatically based on measured data characteristics, maintaining straightforward storage allocation processes while achieving balanced data distribution and eliminating data hotspots.
Data Source
Figure 1~2
Figure 3~4
Figure 5
AI summary
Embodiments of the present disclosure provide a method, apparatus, electronic device and storage medium for storing data. After target data is obtained, a first data feature of the target data is extracted, the first data feature representing a data volume of the target data; the target shard number is obtained based on the first data feature, the target shard number representing the number of storage units for storing target data; a corresponding storage engine is obtained based on the target shard number, and the target data is stored in the storage engines in shards. The matched target shard number is obtained based on the first data feature of the target data, and a corresponding storage engine for storage is obtained based on the target shard number.