File System Engine Integrating Distributed Storage with Object Stores
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern distributed file systems face challenges in seamlessly integrating with cloud-based object stores, as object stores behave differently from traditional file systems, leading to difficulties in accessing and managing data objects across multiple hosts and storage environments.
Innovation Solution
A file system engine is developed to manage the integration of distributed file systems with object stores by determining source buckets, associating them with target directories, applying filter rules, and copying objects based on integration relationships, using cloud computing environments to query and copy objects while maintaining object profiles and indexes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If data is stored in cloud-based object stores, then storage cost is reduced, but file system semantics and data accessibility are lost
Solution Approach 1:
A file system engine acts as an intermediary layer between clients and object stores. The engine translates file system operations (read, write, delete) into object store operations, maintaining file system semantics while utilizing object store infrastructure. The engine manages object buckets, generates object keys from file paths, and handles data synchronization between file systems and object stores.
2Reliability
If data is distributed across multiple hosts and storage environments, then availability and redundancy are improved, but integration complexity increases
Solution Approach 1:
The file system engine provides universal functionality by supporting multiple file system types (NFS, CIFS, POSIX) and operating systems (Linux, Windows, macOS) through a single implementation. It handles diverse storage backends including local disks, network-attached storage, and cloud object stores, unifying access across heterogeneous environments through standardized interfaces.
3Productivity
If object stores are used for long-term storage, then storage efficiency is improved, but access latency increases
Solution Approach 1:
The system segments data management by maintaining active data in local file systems for fast access and archiving inactive data to object stores for efficient long-term storage. The file system engine monitors data access patterns and automatically migrates objects between storage tiers, ensuring hot data remains locally accessible while cold data is stored efficiently in object stores.
Data Source
AI summary
Embodiments are directed to managing data in a file system. A source bucket that contains a plurality of objects on a cloud computing environment may be determined. The source bucket may be associated with a target directory in the file system based on an integration relationship. Object keys from the object store may be provided based on a query and filter rules such that each object key corresponds to an object in the source bucket that conforms to the filter rules. Source objects in the source bucket may be determined based on the object keys such that each source object corresponds to one of the object keys. Target files that correspond to the source objects may be determined based on the integration relationship. In response to determining changed target files or changed source objects, copying the changed source objects or the changed target files to the target directory.