Expandable Probes for Cloud Storage File Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In cloud-based distributed storage systems using consistent hashing, finding and accessing files that have been moved, renamed, or deleted can be time-consuming and lead to system errors due to node shifts caused by server additions or removals, resulting in significant communication and processing overhead when searching all nodes simultaneously, and inefficient file retrieval when traversing the hash ring one node at a time.
Innovation Solution
Implementing a system and method for searching using an expandable set of probes that increase in span exponentially or according to other mathematical orders with each iteration, allowing for a graduated search within the consistent hash structure to locate missing files efficiently without saturating communication or processing bandwidth.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all remaining nodes in the hash structure are searched simultaneously to ensure file location, then the probability of locating the file is maximized, but communication and processing overhead become significant
Solution Approach 1:
The search process is segmented into multiple generations of probes, where each generation searches a specific subset of nodes. Instead of searching all nodes simultaneously, the node set is divided and searched progressively, reducing the communication and processing overhead in each generation while maintaining comprehensive search coverage.
Solution Approach 2:
The probe set dynamically expands across generations. Each generation uses a larger probe set than the previous one, with probes expanding to cover increasingly larger portions of the hash ring. This dynamic expansion allows the system to balance between search thoroughness and overhead by starting with small probe sets and gradually increasing coverage.
2Object-generated harmful factors
If the hash ring is traversed one node at a time, then communication overhead is minimized, but search lag increases and file retrieval becomes inefficient
Solution Approach 1:
The hash ring is segmented into multiple probe sets that can be searched in parallel. Each probe set covers a specific segment of the ring, allowing simultaneous search operations across multiple nodes rather than sequential traversal, thereby reducing search lag while controlling communication overhead through structured probe expansion.
Solution Approach 2:
Each generation of probes searches more nodes than a single-node traversal would examine, using partial parallel search action. The probe sets are designed to cover sufficient portions of the hash ring to achieve efficient search performance without requiring complete examination of all nodes, balancing speed and overhead.
3Device complexity
If a single probe checks the closest adjacent node first, then the search process is simple, but the probability of locating the file decreases when node shifts exceed one position
Solution Approach 1:
The probe set dynamically expands across generations, starting with a single probe at the closest adjacent node and progressively adding probes to cover larger portions of the hash ring. This dynamic expansion maintains simple initial search logic while systematically increasing search coverage to handle node shifts of any magnitude.
Solution Approach 2:
The search process performs preliminary action by first checking the closest adjacent node with a single probe, establishing a simple initial search step. Subsequent generations then systematically expand the probe set to cover additional nodes, ensuring that simple initial logic is combined with comprehensive search capability through predetermined expansion rules.
Data Source
AI summary
Embodiments relate to searching a cloud-based file storage system using a set of expandable probes. Platforms and techniques are provided to search for a file whose location in cloud storage resources is recorded in a consistent hash ring structure. In a hash ring, a file name can generate a hash output corresponding to a position around a circle. The closest clockwise node to that location can contain the file itself, or a link to another location containing or leading to the file. When a link has become obsolete and no longer points to the file, it may be necessary to search for that file. Embodiments perform a graduated search for the missing file, starting one node from the original node, then trying two adjacent nodes, then four adjacent nodes and so forth in expanding fashion, until the file is found or the hash structure is exhausted.