Filesystem Durable Writes to Cloud Object Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional cloud object stores provide weaker consistency guarantees for filesystem operations, such as eventual consistency for write operations, which is inadequate for legacy applications requiring stronger consistency like Read After Write (RAW) or Read After Overwrite.
Innovation Solution
An intermediate layer is introduced between the filesystem layer and the distributed cloud object storage layer, acting as a durable-writer shim to translate read and write operations into get and put operations, and maintaining an index for mapping filesystem content to cloud objects, ensuring stronger consistency through RAW semantics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If cloud object stores use conventional eventual consistency for write operations, then system scalability and performance are improved, but consistency guarantees for legacy applications deteriorate
Solution Approach 1:
The patent introduces an intermediate layer between the filesystem layer and distributed cloud object storage layer. This intermediate layer translates filesystem read/write operations into cloud object store put/get operations while implementing durable write semantics. The intermediary maintains an index mapping filesystem content to cloud objects and ensures Read After Write consistency by coordinating operations across the distributed storage system, thereby preserving legacy application compatibility without compromising filesystem consistency guarantees
2Reliability
If an intermediate layer is introduced to provide stronger consistency, then consistency guarantees for filesystem operations are improved, but system complexity increases
Solution Approach 1:
The patent segments the storage system into distinct layers: a filesystem layer for legacy applications, an intermediate layer for translation and consistency management, and a distributed cloud object storage layer for scalable storage. The intermediate layer is further segmented into components including operation translators, an indexing system, and a durable write implementation. This segmentation allows each layer to be optimized independently while maintaining clear interfaces, reducing overall system complexity despite the added functionality
Data Source
AI summary
Techniques for filesystem durable write operations to cloud object storage are described. In one embodiment, a method includes receiving at least one of a read operation or a write operation from a filesystem layer. The read operation identifies content stored in a distributed content object storage layer that is to be retrieved and the write operation identifies content that is to be stored to the distributed content object storage layer. The method includes converting the read operation into a get operation or the write operation into a put operation. The method further includes storing an index that maps the read or write operation to the get or put operation and includes an association between the content identified by the filesystem layer to objects in the distributed content object storage layer. The method includes sending the get or put operation to the distributed content object storage layer for the identified content.