Service Processing Method for Distributed Object Storage Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed object storage systems face performance bottlenecks and inefficient concurrent processing due to the sort listing method, which results in excessive IO operations and memory usage, especially when dealing with large-scale data and multi-version scenarios.
Innovation Solution
A method that determines optimal parameters for acquiring index shards and objects, assigns weights to these parameters, and lists objects in a matrix format to reduce unnecessary IO operations and improve concurrent processing efficiency by distributing objects across index shards.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If sort listing method is used to acquire objects from multiple shards, then objects can be obtained in sorted order, but the number of IO operations increases significantly to (number of objects * number of shards) / 1000
Solution Approach 1:
The patent divides the listing process into two independent stages: first listing all shards to obtain object keys, then separately acquiring objects based on the collected keys. This segmentation eliminates the need to perform sorting and object acquisition for each shard individually, reducing total IO operations from (objects * shards) / 1000 to approximately objects / 1000 + shards, while still achieving sorted results through a single sorting operation on the collected keys.
Solution Approach 2:
The patent performs preliminary listing of all shards to collect object keys before performing the actual object acquisition. By gathering all necessary key information first and then conducting a single sorting operation followed by unified object acquisition, the system avoids repeated IO operations for each shard while ensuring sorted order is maintained in the final result.
2Productivity
If excessive number of shards is used to improve concurrent capacity, then write operation locking bottlenecks are reduced, but memory usage increases
Solution Approach 1:
The patent introduces a configurable sampling ratio parameter that allows the system to list only a partial portion of objects from each shard rather than all objects. This partial action approach maintains the benefits of multiple shards for concurrency while controlling memory consumption by limiting the number of objects loaded into memory for sorting and processing.
3Measurement precision
If sort listing is performed across multiple shards, then objects are returned in sorted order, but the data size acquired becomes 1000 * number of shards times the required data size
Solution Approach 1:
The patent extracts only the necessary object keys from each shard during the preliminary listing phase, rather than acquiring full object data for each shard. After collecting all keys and performing a single sorting operation, the system then acquires objects based on the sorted keys. This extraction approach ensures sorted order is achieved while minimizing data size acquired to approximately 1000 times the required size plus the small overhead of storing shard metadata.
Data Source
AI summary
The disclosure discloses a method for service processing and system, including: determining a first quantity parameter for acquiring index shards each time and a second quantity parameter of objects acquired by each of the index shards each time; assigning first weight to the first quantity parameter to obtain third quantity parameter for acquiring the index shards; listing, from a plurality of index shards at a bucket, index shards corresponding to the third quantity parameter, and listing, from each of the index shards which is listed, the objects corresponding to the second quantity parameter, so as to obtain a matrix which takes each index shard as a column and a plurality of objects corresponding to each shard as rows; successively extracting the plurality of the objects corresponding to each row of the matrix; and according to preset number of concurrent processing, processing the plurality of the objects extracted each time.