Virtual File System for Small-File Storage in Kubernetes

The patent segments the file system into two distinct layers: a virtual file system layer for managing small files and a physical distributed file system layer (Lustre) for storing large files. This segmentation allows each layer to be optimized independently - the virtual file system can implement efficient metadata management for small files while the underlying Lustre system maintains its high storage capacity and throughput for large files, thereby resolving the contradiction between storage capacity and read-write efficiency.

The patent introduces a virtual file system as an intermediary layer between the user-mode application and the physical distributed file system. This intermediary handles metadata operations for small files efficiently, preventing the metadata management bottleneck from affecting the underlying storage system's performance. The virtual file system acts as a buffer that translates small file operations into efficient storage operations on the Lustre system, maintaining both high storage capacity and read-write efficiency.

The patent segments metadata management into two parts: a virtual metadata layer for small files that provides efficient access control and retrieval, and a physical metadata layer for the distributed file system. The virtual metadata layer maintains indexes and metadata structures specifically optimized for small file operations, allowing fast file access without requiring the entire distributed file system to process metadata requests, thus preserving storage performance while enabling ease of operation.

The virtual file system serves as an intermediary that handles all metadata operations for small files before they reach the physical distributed file system. This intermediary layer caches and manages metadata locally, enabling fast file access through efficient metadata retrieval without burdening the entire file system. The intermediary filters and processes metadata requests, ensuring that storage performance is not degraded by the large number of small files.

The patent implements a universal virtual file block structure that can store both small files and serve as a bridge to the distributed file system. This multi-functional design allows the same virtual file block mechanism to handle metadata storage, small file data, and coordination with the underlying Lustre system, thereby improving read-write performance without proportionally increasing device complexity. The universal structure consolidates multiple functions into a single component, reducing the overall complexity overhead.

The patent employs a nested structure where virtual file blocks are contained within the virtual file system layer, which itself is nested within the container orchestration environment, and finally within the distributed file system infrastructure. This nesting allows each layer to manage its own complexity independently while benefiting from the layers below. The virtual file blocks provide improved read-write performance through their specialized structure, while the nested architecture contains and manages the resulting complexity at appropriate abstraction levels.