Multi-tiered Metadata Mapping for Distributed Storage Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed computing environments, accessing metadata for virtual disk data is inefficient due to the distribution of metadata across multiple nodes, leading to increased latency and resource consumption, which negatively impacts overall data access performance.
Innovation Solution
Implementing a multi-tiered metadata mapping regime where a metadata virtual disk is collocated with the corresponding data virtual disk, using a distributed metadata database to map logical representations to physical storage locations, and storing metadata in in-memory data structures for low-latency access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If metadata is distributed across multiple nodes of the computing cluster, then data access scalability is improved, but data access latency increases due to multiple hops
Solution Approach 1:
The patent segments metadata into two distinct types: virtual disk metadata stored locally at each computing node, and file metadata stored in the distributed metadata database. This segmentation allows frequently accessed virtual disk metadata to be retrieved locally without network hops, while file metadata remains distributed for scalability. The segmentation resolves the contradiction by eliminating latency-critical metadata from the distributed storage path.
Solution Approach 2:
The patent performs preliminary action by pre-creating virtual disk metadata at each computing node when a virtual disk is created or migrated. This virtual disk metadata includes pre-calculated information about extents and their locations. When data access is needed, this pre-prepared metadata is immediately available locally, eliminating the need for runtime metadata retrieval from other nodes and thus reducing access latency.
2Productivity
If metadata is stored in distributed metadata database, then metadata management scalability is improved, but resource consumption increases due to network traffic and inter-node communication
Solution Approach 1:
The patent extracts virtual disk metadata from the distributed metadata database and stores it locally at each computing node. This extraction eliminates the need to repeatedly query the distributed database for virtual disk metadata, significantly reducing network traffic and inter-node communication. Only file-level metadata operations require distributed database access, while virtual disk operations use local metadata, thus reducing resource consumption.
Solution Approach 2:
Each computing node maintains its own virtual disk metadata locally, enabling self-service for metadata operations. When a virtual disk is created or migrated to a node, the node automatically creates and stores the virtual disk metadata locally without requiring continuous communication with other nodes. This self-service approach minimizes network dependency and resource consumption for routine operations.
3Adaptability or versatility
If virtual disk metadata is accessed from remote nodes, then data distribution flexibility is improved, but processing power consumption increases due to additional lookup operations
Solution Approach 1:
The patent performs preliminary action by pre-computing and storing virtual disk metadata at each computing node when virtual disks are created or migrated. This pre-computed metadata includes all necessary information about extents, their locations, and mapping relationships. When data access is required, the node can immediately use this pre-prepared metadata without performing additional lookup operations, thus reducing processing power consumption while maintaining data distribution flexibility.
4Productivity
If multiple levels of metadata are implemented, then data access efficiency is improved, but device complexity increases
Solution Approach 1:
The patent segments metadata into two distinct types with clearly defined responsibilities: virtual disk metadata stored locally at each computing node, and file metadata stored in the distributed metadata database. This segmentation creates a simple, hierarchical architecture where each layer serves a specific purpose. The virtual disk metadata layer handles fast local access for virtual disk operations, while the file metadata layer handles distributed file-level operations, thus improving efficiency without creating complex interdependencies.
Data Source
AI summary
Systems for low-latency data access in distributed computing systems. A method embodiment commences upon generating a first storage area in local storage of a first computing node. Access to the first storage area is provided through the first computing node. A second storage area is generated wherein the second storage area comprises a first set of metadata that comprises local storage device locations of at least some of the local storage areas of the first storage area. A set of physical access locations of the second storage area is stored to a database that manages updates to the second set of metadata pertaining to the second storage area. Accesses to the first storage area are accomplished by querying the database to retrieve a location of the second set of metadata, and then accessing the first storage area through one or more additional levels of metadata that are node-wise collocated.