Partition View Segmentation for Distributed Storage Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional centralized storage systems face inefficiencies in read/write operations and resource utilization when managing ultra-large-scale data, particularly due to excessive complexity in partition views and resource occupation in distributed storage systems with a large number of storage disks.
Innovation Solution
A data processing method and apparatus that optimizes partition management by calculating and managing IDs for key-value data, reducing the number of partitions per storage disk, and dynamically adjusting partition allocation to match the number of storage disks, thereby reducing system resource consumption and improving data forwarding efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If distributed storage system uses a large number of storage disks, then storage capacity and scalability are improved, but partition view complexity and system resource occupation increase excessively
Solution Approach 1:
The patent segments the partition view into multiple sub-partition views, where each sub-partition view manages a specific range of partition IDs. This segmentation allows the system to handle large numbers of storage disks while keeping each sub-partition view manageable in size, thus reducing the complexity of individual partition views while maintaining overall storage capacity.
2Manufacturing precision
If each storage disk has excessive partitions, then data distribution uniformity is improved, but data packet forwarding efficiency deteriorates due to complex partition view lookup
Solution Approach 1:
The patent segments the partition view into multiple sub-partition views, each managing a specific range of partition IDs. This allows the system to maintain fine-grained partitioning for uniform data distribution while enabling efficient forwarding by limiting the search space to a specific sub-partition view based on the destination partition ID range.
Solution Approach 2:
The patent pre-calculates and determines which sub-partition view contains the target partition before performing the actual partition view lookup. This preliminary action narrows down the search scope significantly, improving forwarding efficiency while maintaining uniform data distribution across partitions.
3Speed
If partition view information is stored in memory, then access speed is improved, but system memory resource occupation increases
Solution Approach 1:
The patent segments the partition view into multiple sub-partition views that are stored in memory, while the overall partition view structure remains on disk. This segmentation allows frequently accessed sub-partition views to be cached in memory for fast access, while the complete partition view information can be reconstructed from disk when needed, thus balancing memory usage with access speed.
Data Source
AI summary
A data processing technology is provided, and is applied to a partition management device. The partition management device stores a partition view, the partition view records a correspondence between an ID of a current partition and an address of a storage disk, and a total quantity of current partitions may be less than a total quantity of final partitions.