User-Space File System for Network Storage Integration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed file storage systems face inefficiencies when storage capacity on data nodes becomes filled with less relevant data, leading to wasted processing and networking capabilities, and adding more data nodes is not a desirable solution due to increased equipment and space requirements, as well as hardware configuration limitations.
Innovation Solution
Implementing a user-space file system that intercepts file system level commands to present external network storage as a virtual local storage volume, using preconfigured metadata to prevent false failure detection during transient communication failures, thus avoiding cascading data migration and maintaining system availability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If more data nodes are added to maintain storage capacity and processing capacity, then storage system capacity is improved, but equipment requirements, cooling requirements, networking requirements, and floor space requirements increase
Solution Approach 1:
The data node is designed to perform multiple functions: it can store data locally on direct attached storage devices and simultaneously access network storage through the user-space file system. This multi-functionality allows the system to maintain storage capacity without adding new data nodes, as existing nodes can serve both local and networked storage roles.
Solution Approach 2:
A user-space file system acts as an intermediary layer between the distributed file system and network storage. This intermediary enables data nodes to access network storage as if it were local direct attached storage, effectively increasing available storage capacity without requiring additional physical data nodes or their associated infrastructure.
2Quantity of substance
If data is permanently removed from data nodes to free up storage capacity, then storage space is recovered, but processing and networking capabilities are left unused and wasted
Solution Approach 1:
The system enables data nodes to maintain full functionality by allowing them to keep data on direct attached storage while also accessing network storage through the user-space file system. This prevents the waste of processing and networking capabilities that would occur if data were permanently removed, as nodes continue to operate at full capacity.
3Reliability
If device check process performs thorough failure tests by sending file system level requests, then storage device reliability is improved, but transient communication failures cause false failure detection and cascading data migration
Solution Approach 1:
The user-space file system serves as an intermediary that intercepts file system level requests from the device check process. When a transient communication failure occurs, the intermediary prevents the failure from propagating to the device check process, thereby avoiding false failure detection while maintaining accurate reliability monitoring for actual storage device failures.
Solution Approach 2:
The system converts the potentially harmful effect of transient communication failures into a benefit by using the user-space file system to filter and prevent these transient failures from triggering false device check failures. This prevents unnecessary cascading data migration while still maintaining reliable detection of actual storage device failures.
Data Source
AI summary
In some examples, a data node may provide a file system executed in a user space. The data node may invoke a device check process configured to perform a test for failure of a storage device associated with a target volume by sending a file system level request for obtaining metadata of data stored to the target volume. For instance, the target volume may be a virtual storage volume that represents storage capacity on a network storage. The user-space file system may receive the request for the metadata and may retrieve preconfigured metadata of the target volume that is stored locally in advance of the invoking of the device check process. The preconfigured metadata may be sent to the device check process in response to the request. For example, the preconfigured metadata may at least partially cause the target volume to pass the test.