Parallel Key Listing in Distributed File Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed computing systems, retrieving a list of data objects across multiple servers can be slow and difficult to parallelize, especially as storage demands increase, due to the challenge of efficiently tracking and listing data objects in a distributed processing framework file system.
Innovation Solution
A system and method for parallelizing the listing of data objects by using an application programming interface to receive requests for listings of keys, determining the number of threads based on the obtained set of keys, and splitting them into subsets for parallel processing, where each thread generates a sub-listing of keys and merges them into a complete listing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data objects are stored across multiple servers in a distributed system, then storage capacity and durability are improved, but the speed and efficiency of retrieving and listing data objects deteriorates
Solution Approach 1:
The patent divides the distributed file system into hierarchical directories and partitions data objects across multiple servers within each directory. The listing operation is segmented into parallel tasks that can be executed independently on different server partitions simultaneously, resolving the contradiction between distributed storage and listing speed.
Solution Approach 2:
The patent introduces a new dimensional approach by implementing a hierarchical directory structure with multiple levels (root directory, subdirectories, etc.). This adds a spatial dimension to the listing operation, allowing the system to traverse and list data objects in an organized manner across the distributed architecture, improving both scalability and speed.
2Quantity of substance
If the number of data objects increases to meet storage demands, then storage capacity is improved, but the difficulty and time required for tracking and listing data objects increases
Solution Approach 1:
The patent implements preliminary indexing and metadata generation when data objects are stored in the distributed file system. This preliminary action creates an organized structure with directory information and object metadata that can be quickly queried and traversed, significantly reducing the time required for listing operations even as the number of data objects grows to petabyte scales.
3Ease of manufacture
If a traditional sequential listing approach is used in a distributed file system, then implementation simplicity is maintained, but processing efficiency and parallelization capability deteriorates
Solution Approach 1:
The patent implements a dynamic listing approach where the file system can adaptively choose between sequential and parallel listing methods based on the size of the directory, the number of data objects, and system resources available. This dynamic behavior allows the system to maintain simplicity for small directories while achieving high processing efficiency for large-scale distributed storage scenarios.
Data Source
AI summary
A system and method for improving the speed of generating a list of previously-uncounted items stored with a computing resource service provider. The system and method involve obtaining a set of keys from a data store, wherein each key of the set of keys corresponds to an item in a group of items, wherein a quantity of items in the group is uncounted. The system and method further includes generating a first sub-listing of keys based at least in part on a first key range of the set of keys by executing a first thread, generating a second sub-listing of keys based at least in part on a second key range of the set of keys by executing a second thread, combining the first sub-listing of keys with the second sub-listing of keys to produce a list of keys, and providing the list of keys.