Hierarchical Namespace for Scalable Distributed Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed or clustered file systems have limitations on the number of simultaneous clients they can support, which restricts the scalability of shared storage resources, even when additional commodity storage is added, as they are designed with inherent limitations that plateau the utility of a highly scalable datastore.
Innovation Solution
A method is introduced that exposes a hierarchical namespace to clients while internally using a flat namespace to track objects, allowing for the creation of top-level objects representing separate storage devices, enabling scalability by distributing different parts of the clustered file system across various objects, thus overcoming limitations on the number of simultaneous clients.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If current distributed or clustered file systems are used to manage the datastore, then concurrency control mechanisms are provided, but the number of simultaneous clients is limited
Solution Approach 1:
The patent divides the file system into multiple independent file system objects, each capable of handling client requests independently. This segmentation allows the system to scale horizontally by adding more file system objects, thereby increasing the number of simultaneous clients without being constrained by the limitations of a single centralized file system.
Solution Approach 2:
The patent introduces a hierarchical namespace with top-level objects that represent separate file system abstractions. This adds a dimensional layer to the storage architecture, allowing clients to access different file system objects through the hierarchical structure, thereby enabling scalability beyond the client limits of individual file systems.
2Quantity of substance
If additional commodity storage is added to increase datastore capacity, then storage scalability is improved, but the utility is plateaued due to client access limitations
Solution Approach 1:
By segmenting the storage into multiple file system objects, each object can independently utilize additional commodity storage. This allows the system to scale storage capacity without being constrained by client access limitations of a single file system, as new storage can be added to any file system object.
Solution Approach 2:
The hierarchical namespace provides a universal access interface that works across multiple file system objects. Clients can access storage resources through the unified hierarchical structure, making the additional commodity storage utility regardless of which underlying file system object the data resides in.
3Productivity
If a flat namespace is used to store and retrieve objects, then storage and retrieval efficiency is improved, but flexibility to create hierarchical relationships is lost
Solution Approach 1:
The patent introduces a hierarchical dimension to the namespace by creating top-level objects that represent separate file system abstractions. This hierarchical structure is layered on top of the flat namespace implementation, allowing clients to access objects through hierarchical paths while the underlying storage maintains efficient flat namespace operations.
Solution Approach 2:
The patent introduces file system objects as intermediary layers between the flat namespace storage and the clients. These intermediaries provide the hierarchical organization and namespace management functionality, while the underlying flat namespace continues to provide efficient storage and retrieval operations.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Techniques are disclosed for providing a file system interface for an object store intended to support simultaneous access to objects stored in the object store by multiple clients. In accordance with one method, an abstraction of a root directory to a hierarchical namespace for the object store is exposed to clients. The object store is backed by a plurality of physical storage devices housed in or directly attached to the plurality of host computers and internally tracks its stored objects using a flat namespace that maps unique identifiers to the stored objects. The creation of top-level objects appearing as subdirectories of the root directory is enabled, wherein each top-level object represents a separate abstraction of a storage device having a separate namespace that can be organized in accordance with any designated file system.