Virtual Directory Namespace for Unstructured Storage Reliability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed data storage systems face challenges in maintaining data across multiple geographically disparate storage nodes in a simple, robust, and reliable manner.
Innovation Solution
A method is implemented in a data storage system where requests for data items are sent to multiple storage nodes using multicast and unicast communications, utilizing a virtual directory structure with unique keys to access and manage data items across a network of interconnected storage nodes, allowing for efficient data retrieval, storage, and deletion through an Application Programming Interface (API).
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored across multiple geographically disparate storage nodes, then data availability and reliability are improved, but system complexity and difficulty of maintenance increase
Solution Approach 1:
The patent introduces a namespace service as an intermediary layer between clients and storage nodes. This service handles namespace resolution, data location tracking, and coordination of operations across distributed nodes. By centralizing these control functions, the system achieves high data reliability through geographic distribution while maintaining manageable complexity through a unified coordination interface.
Solution Approach 2:
The system segments data storage across multiple independent storage nodes distributed geographically, while maintaining a separate namespace service layer. This segmentation allows each storage node to operate independently for reliability, while the namespace layer provides unified management to control complexity.
2Productivity
If a virtual directory structure is implemented on unstructured storage, then data access efficiency is improved, but the complexity of data management increases
Solution Approach 1:
The namespace service acts as an intermediary that translates virtual directory structure operations into unstructured storage operations. It maintains namespace metadata separately from actual data, allowing efficient virtual directory access while keeping the underlying storage simple and unstructured. This separation of concerns improves access efficiency without proportionally increasing management complexity.
Solution Approach 2:
The patent adds a namespace dimension above the physical storage layer. The virtual directory structure exists in this abstract namespace dimension, while actual data resides in the physical unstructured storage dimension. This dimensional separation allows efficient hierarchical access patterns without imposing corresponding structural complexity on the storage layer.
3Speed
If multicast and unicast communications are used for data requests, then data retrieval speed is improved, but network resource consumption increases
Solution Approach 1:
The system uses multicast for namespace resolution requests where multiple nodes need to be queried, but limits multicast usage to only when necessary. For direct data retrieval, unicast is used to target specific storage nodes. This partial use of multicast provides speed benefits when needed while avoiding excessive network resource consumption through selective application of the communication method.
Data Source
AI summary
An exemplary method includes receiving a request to store a data item within a directory of a virtual file structure layer for an unstructured storage layer of a data storage system, caching an identifier of a collection object representative of the directory, sending a query to a plurality of storage nodes included in a cluster of the data storage system, receiving, in response to the query, replies from a subset of the storage nodes that are available to store the data item, selecting, based on the replies, a storage node from the subset of storage nodes, transmitting the data item to the selected storage node for storage by the selected storage node within the unstructured storage layer, accessing, based on the cached identifier, the collection object, and updating the collection object to include a reference to the data item stored in the selected storage node.