Acquired Namespace Metadata Service for Distributed Storage Expansion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current HDFS systems face limitations in expanding storage capacity and optimizing data distribution across multiple namespaces, leading to disruptive data transfers and inefficient capacity utilization.
Innovation Solution
A system and method that utilize an acquired namespace metadata service to manage and control storage resources, allowing for the addition of storage resources, merging of namespaces, and transparent data migration between storage nodes, enabling flexible and efficient storage management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If HDFS v1 is used to manage storage resources, then the system provides simple namespace management, but the system cannot expand storage capacity beyond the original data stores
Solution Approach 1:
The patent segments the namespace management function by introducing an acquired namespace metadata service that operates independently from the original HDFS namespace. This allows the first file system to acquire and manage additional namespaces from the distributed file system without merging them, enabling storage expansion while maintaining separate namespace management for each segment
Solution Approach 2:
The acquired namespace metadata service acts as an intermediary between the first file system and the storage nodes of the distributed file system. It enables the first file system to control access to storage resources without directly managing the underlying HDFS infrastructure, thus expanding capacity while managing complexity through an intermediate layer
2Productivity
If HDFS v2 is used to support multiple namespaces, then the system allows simultaneous use of multiple HDFS stores, but there is no efficient way to balance and tier data between different stores
Solution Approach 1:
The patent applies local quality by allowing different namespaces to have different characteristics and management policies. The first file system can acquire specific namespaces from the distributed file system and manage them with localized control, enabling efficient data balancing and tiering within each namespace while maintaining overall system coordination
Solution Approach 2:
The system enables dynamic data distribution across multiple namespaces through the acquired namespace metadata service. Data can be balanced and tiered between different acquired namespaces based on performance and capacity requirements, with the metadata service dynamically adjusting namespace management as storage conditions change
3Quantity of substance
If additional data stores are added to expand storage capacity, then the system increases storage capacity, but large amounts of data must be transferred which is disruptive and time consuming
Solution Approach 1:
The patent uses copying rather than moving data when expanding storage capacity. The first file system acquires namespaces from the distributed file system, creating additional namespace references to the same storage resources. This allows storage capacity expansion without requiring data transfer, as the acquired namespaces provide alternative access paths to the same data
4Adaptability or versatility
If the original data store is abandoned to use a new data store, then the system can use new storage resources, but the process is disruptive and time consuming
Solution Approach 1:
The patent merges the namespace management of the first file system with the acquired namespaces from the distributed file system. Instead of abandoning the original data store, the system combines access to both original and acquired namespaces through the acquired namespace metadata service, enabling flexible storage resource usage without disruptive transitions
Data Source
AI summary
Systems and methods are provided for adopting and controlling storage resources of a distributed file system using an acquired namespace metadata service. For example, a computing system includes a first file system, and a distributed file system, which is separate from the first file system. The distributed file system includes storage nodes for storing data. The first file system includes an acquired namespace metadata server that is configured to execute on one or more nodes of the first file system. To adopt and control storage resources of the distributed file system, the first file system acquires a namespace of the distributed file system and uses the acquired namespace metadata server to manage the acquired namespace of the distributed file system. Moreover, the first file system uses the acquired namespace metadata server to directly communicate with and control access to the storage nodes of the distributed file system.