Hierarchical Namespace Service for Cloud Blob Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud storage systems often lack a hierarchical namespace structure, which is essential for big data and data analytics applications, leading to increased cost and complexity due to the need for data silos and frequent renaming operations, and existing solutions fail to scale horizontally.
Innovation Solution
Implementing a hierarchical namespace service that maps hierarchical file system commands to a flat namespace, enabling both flat and hierarchical namespace access over the same underlying object storage layer, with support for single-node and multi-node partition transactions, ensuring strong consistency and horizontal scalability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If a flat storage structure is used, then storage simplicity and accessibility are improved, but hierarchical organization capability and file system operations deteriorate
Solution Approach 1:
The patent segments the namespace into hierarchical levels (directories, subdirectories, files) while maintaining a flat underlying storage structure. The virtual file system layer segments access paths into components that can be independently managed, enabling hierarchical operations without physically organizing data in a hierarchical manner.
Solution Approach 2:
The patent introduces a virtual file system as an intermediary layer between the flat storage structure and user applications. This virtual layer mediates between the simple flat storage and complex hierarchical access requirements, translating file system operations into flat storage operations while preserving hierarchical semantics.
2Adaptability or versatility
If data silos are created for different use cases, then application-specific data management is improved, but system complexity and cost increase
Solution Approach 1:
The patent implements a universal flat storage structure that serves multiple purposes through a single system. The same storage infrastructure supports different use cases (analytics, backup, archiving) without requiring separate data silos, achieving multi-functionality while reducing overall system complexity.
Solution Approach 2:
The patent merges previously separate data management systems into a unified flat storage structure. By combining multiple use cases into a single storage system with a common access layer, the patent eliminates the need for multiple data silos and their associated management overhead.
3Adaptability or versatility
If frequent renaming operations are performed, then data reorganization flexibility is improved, but processor operations and time consumption increase
Solution Approach 1:
The patent uses virtual names as copies or aliases that map to the same physical storage location. Instead of moving data during renaming operations, the system creates new virtual name entries that point to the same underlying object, making renaming operations instantaneous without data copying or movement.
Solution Approach 2:
The patent implements dynamic name resolution where virtual names can be created, deleted, or modified without affecting the underlying physical storage. The mapping between virtual and physical names is dynamic and can be changed at any time, enabling flexible reorganization without operational overhead.
4Adaptability or versatility
If hierarchical namespace service is implemented, then hierarchical data organization is improved, but system complexity increases
Solution Approach 1:
The patent implements a nested structure where virtual file system components (directories, files) are nested within each other to form hierarchical paths. This nesting is achieved through the virtual name layer without requiring physical nesting of storage structures, enabling hierarchical organization with minimal additional complexity.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Methods, systems, and computer programs are presented for providing file system functions on a cloud storage system based on blob storage. A cloud storage system comprises storage clusters, and each storage cluster comprises a plurality of storage nodes (each including a partition layer that stores data in blobs organized within containers, and a stream layer for streaming blob data), a hierarchical namespace service for implementing file system functions on the data stored in the blobs and, a front end layer for processing user requests to access and manage data stored on the blobs. The file system functions include flat namespace functions and hierarchical namespace functions. Further, the front end layer accesses the partition layer to access the blobs based on addresses of blobs and accesses the hierarchical namespace service to identify the blobs.