Key Value File System Offloading Metadata Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional file systems face inefficiencies in data block management and defragmentation, leading to high CPU usage and multiple I/O operations when handling large directories and files, which affects performance and I/O efficiency.
Innovation Solution
A key-value file system (KVFS) that represents files and directories as inodes with key-value pairs, offloading data block management to a KV store, allowing single I/O operations for lookups and truncations, and using containers to manage data blocks of fixed sizes, reducing the need for explicit defragmentation and metadata management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If conventional file systems use multiple levels of metadata to manage data blocks in large directories, then data organization is achieved, but multiple I/O operations are required for lookups reducing performance
Solution Approach 1:
The patent extracts the metadata management functionality from the conventional file system and relocates it to a key-value store. The key-value store assumes responsibility for storing and retrieving directory entries and file metadata, allowing the file system to perform lookups by directly querying the key-value store with directory paths as keys, thereby eliminating multiple I/O operations through the traditional metadata hierarchy.
Solution Approach 2:
The key-value store acts as an intermediary between the file system and the underlying storage device. It provides a simplified interface for metadata operations, where directory paths are stored as keys and their corresponding data block locations as values, mediating the complex metadata navigation that would otherwise require multiple I/O operations.
2Productivity
If conventional file systems perform data defragmentation to improve I/O performance, then data access efficiency is improved, but significant host CPU cycles are consumed
Solution Approach 1:
The key-value store performs defragmentation autonomously without requiring host CPU intervention. The system allows the key-value store to manage its own data block allocation and defragmentation internally, using its own resources rather than consuming host CPU cycles. This self-service approach maintains I/O performance while eliminating the computational burden on the host system.
3Reliability
If the file system makes several round trips to allocate data blocks and metadata in cache, then data storage is achieved, but I/O efficiency is reduced
Solution Approach 1:
The patent merges the allocation of data blocks and metadata into a single I/O operation by utilizing the key-value store's unified storage structure. When data needs to be stored, the system performs one I/O operation to write both the data and its associated metadata (stored as key-value pairs) to the key-value store simultaneously, eliminating the need for separate round trips to allocate and store data and metadata.
4Ease of operation
If the file system traverses all metadata of a large directory to lookup an entry, then complete directory search is achieved, but execution of multiple I/O operations reduces performance
Solution Approach 1:
The key-value store pre-organizes directory entries by storing directory paths as keys and their corresponding data block locations as values. This preliminary organization allows the file system to perform lookups by directly querying the key-value store with the directory path, retrieving the required information in a single operation without traversing through multiple levels of metadata structures.
Data Source
AI summary
A file system includes: an application programming interface (API) configured to provide a file system access to an application running on a host computer; a key value file system configured to represent a file or a directory as an inode including one or more key-value pairs; a virtual file system configured to direct a file system call received from the application to the key value file system; and a key value API configured to provide the file system access to data stored in a data storage device. Each key-value pair contained in the inode includes a name of the file or the directory as a key and an identifier of a container that is associated with the file or the directory as a value. The data of the file is stored in the data storage device as being divided into one or more data blocks of a fixed size, and each of the one or more data blocks associated with the data of the file is accessible within the key value file system using the one or more key-value pairs.