Lightweight Filesystem Cache for Remote Immutable Data Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern general-purpose filesystems incur overhead for database systems that operate on large, immutable files stored remotely, as they maintain unnecessary data structures like inodes and journals, affecting performance.
Innovation Solution
A lightweight filesystem that divides local storage into data blocks with metadata in the last portion, eliminating directory hierarchies and using remote data store identifiers, allowing metadata to be memory-efficient and cacheable.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a general-purpose filesystem is used to manage remote storage, then data access functionality is provided, but computing resource usage increases due to overhead from maintaining inodes, journals, and directory hierarchies
Solution Approach 1:
The patent extracts and removes unnecessary filesystem components (inodes, journals, complex directory hierarchies) from the system, keeping only the essential metadata needed for remote storage access. This eliminates the overhead of maintaining general-purpose filesystem structures while preserving core data access functionality.
Solution Approach 2:
Instead of using a full filesystem structure to manage remote storage, the patent inverts the approach by using a simplified metadata structure that directly maps to remote storage objects, eliminating the need for traditional filesystem abstractions.
2Adaptability or versatility
If a general-purpose filesystem with directory hierarchies is used, then file organization capability is provided, but device complexity increases due to unnecessary data structures
Solution Approach 1:
The patent removes complex directory hierarchy structures and replaces them with a flat metadata structure that directly references remote storage objects, eliminating unnecessary organizational layers while maintaining file access capability.
Solution Approach 2:
The patent segments the filesystem into minimal essential components only - primarily metadata structures that reference remote storage - eliminating the need for complex hierarchical organization while preserving basic file access functionality.
3Reliability
If sophisticated data structures like inodes and journals are maintained, then crash recovery and access control features are provided, but access performance deteriorates due to overhead
Solution Approach 1:
The patent extracts and removes journaling structures and complex inode maintenance mechanisms, relying instead on the simplicity of the metadata structure and remote storage properties to provide adequate recovery capability without the performance overhead of traditional filesystem safeguards.
Data Source
AI summary
A lightweight filesystem may be provided for remote storage caching. A filesystem may maintain a persistent cache for a data set stored as data files in immutable data objects in a remote data store. Filesystem metadata may be evaluated to determine whether a portion of a data file is stored in the persistent cache according to an offset and length specified in a request. If in the persistent cache, data obtained from a data block in the persistent cache may be returned. If not in the persistent cache, then the remote data store may be accessed and the data file in the immutable data object read to obtain the portion of the data file.