Layered Metadata Lookup in Distributed Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed storage systems, the existing mechanisms for looking up metadata in edge scenarios are inefficient due to the need for multiple hops across physically distant nodes, leading to high latency and prolonged search times, especially in applications like self-driving vehicles where data continuity requires rapid access.
Innovation Solution
A method that utilizes a layered structure to quickly locate metadata by determining a data identifier with layer information, calculating the distance between node identifiers, and identifying a second node for metadata acquisition, thereby reducing the need for extensive node traversal and improving data processing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If distributed hash tables are used to look up metadata in a decentralized distributed storage system, then the system can provide high throughput and high availability, but the lookup process requires traversing search paths or going through multiple hops, resulting in high latency and prolonged search times
Solution Approach 1:
The patent segments the distributed storage system into multiple layers based on physical distance, with each layer containing nodes at similar distances from the client. This segmentation allows the lookup process to focus first on nearby nodes (layer 1), then progressively search farther layers if needed, rather than uniformly traversing all nodes in a decentralized system. The segmentation resolves the contradiction by organizing nodes spatially to reduce average search distance while maintaining system-wide availability.
Solution Approach 2:
The patent implements local quality by prioritizing lookup operations within the local layer (nodes at minimal physical distance) before expanding to remote layers. Metadata is stored with awareness of its physical location, and lookups preferentially access locally-stored metadata first. This local-first approach reduces search time for the majority of operations while preserving the ability to access remote metadata when necessary, thus improving speed without sacrificing reliability.
2Reliability
If metadata is stored across multiple distant nodes in a decentralized system, then data availability is improved, but the physical distance between nodes increases, leading to higher latency in metadata retrieval
Solution Approach 1:
The patent introduces a spatial dimension (physical distance) as an additional organizing principle for metadata storage, creating a layered structure that complements the logical decentralized distribution. Instead of purely random or hash-based distribution across distant nodes, metadata is organized by physical proximity layers. This dimensional addition allows the system to maintain decentralized availability while optimizing retrieval speed through spatial awareness, resolving the contradiction between distribution and speed.
Solution Approach 2:
The patent performs preliminary action by pre-organizing metadata storage locations according to their physical distance layers before actual data access occurs. When data is stored, its metadata is placed in appropriate layers based on predicted access patterns and physical location. This preliminary spatial organization enables faster retrieval by ensuring that frequently accessed metadata resides in nearby layers, reducing latency without requiring ad-hoc search optimization during actual access operations.
Data Source
AI summary
Embodiments of the present disclosure relate to a method, an electronic device, and a computer program product for looking up data. This method includes: acquiring, at a first node, a data identifier for data to be looked up, wherein the first node is located in a system having a layered structure, and the data identifier comprises layer information related to a layer in the layered structure. The method further includes: determining a distance between the data identifier and a node identifier for the first node. The method further includes: determining, based on the distance, a second node that can be used to acquire metadata for the data for acquiring the metadata, wherein the distance is related to positions of the first node and the second node in the layered structure. The method further includes: determining, based on the metadata, a third node storing the data for acquiring the data.