Remote File Service Layer for POSIX Operations on Cloud Object Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud object storage systems lack support for certain POSIX-style file operations, leading to high latencies and inefficiencies in managing recovery data, making them sub-optimal for applications requiring frequent file updates and recoverability.
Innovation Solution
A remote file service (RFS) system acts as an interface for cloud object storage, executing POSIX-style file operations locally and recording these operations in journal files, which are then stored in cloud object storage, allowing for efficient recovery of application states even when the local system fails.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If cloud object storage is used for storing application files, then reliability and accessibility are improved, but POSIX-style file operations are not supported and latency increases
Solution Approach 1:
A remote file service (RFS) system is introduced as an intermediary layer between the application and cloud object storage. The RFS system translates POSIX-style file operations into cloud object storage operations, enabling applications to use familiar file operation interfaces while leveraging the reliability of cloud storage. The RFS system maintains a file system namespace and handles file operations locally, then synchronizes with cloud object storage.
2Adaptability or versatility
If cloud object storage is used for storing recovery data, then accessibility from any machine is improved, but write operations become slow due to high latencies
Solution Approach 1:
The system segments storage operations into two parts: frequent write operations are performed locally on the RFS system for immediate speed, while cloud object storage serves as the backend for durability and remote accessibility. The RFS system maintains local copies of data and synchronizes with cloud storage, allowing fast local writes while preserving eventual consistency and remote access capabilities.
3Stability of the object's composition
If cloud object storage is used without local caching, then storage consistency is improved, but read and write operations experience high latency
Solution Approach 1:
The RFS system performs preliminary actions by maintaining local caches and file system namespaces before cloud object storage operations are needed. Data is pre-loaded and kept locally on the RFS system, allowing immediate read and write operations without waiting for cloud storage latency. The local state is then synchronized with cloud object storage to maintain consistency.
4Productivity
If POSIX-style file operations are implemented on cloud object storage directly, then operational efficiency is improved, but cloud object storage performance deteriorates due to unsupported operations
Solution Approach 1:
The RFS system acts as a mediator that implements POSIX-style file operations locally without directly burdening cloud object storage. The RFS system handles file creation, deletion, renaming, and other POSIX operations in its local file system namespace, then translates necessary operations to cloud object storage APIs. This preserves cloud object storage performance while providing efficient POSIX-style operations to applications.
Data Source
AI summary
A method implemented by data processing systems enables usage of a cloud object storage for storing data accessed based on execution of file operations that are unsupported by the cloud object storage. The method includes executing an application configured to access application data; as the application executes, generating instructions to perform operations, unsupported by cloud object storage, to modify the application data; transmitting the instructions to a first file service system representing a location for the application data to be stored in the cloud object storage; recording an indication of the operations specified by the instructions; and transmitting, to the cloud object storage, the indication of the operations specified by the instructions, whereby transmittal of the indication to cloud object storage enables a second data processing system to access the application data that is modified in accordance with the operations unsupported by cloud object storage.