Volatile Memory File Object Durability via Log Replay
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face challenges in maintaining durability of file objects in volatile system memory, particularly during power failures or reboots, leading to data loss and the need for clients to resend file system operations, which consumes resources and increases network bandwidth usage.
Innovation Solution
Implementing a file system log in an object storage system that logs file system operations and allows for the replay of these operations to restore file objects in volatile memory, even in the absence of local persistent storage access, thereby ensuring data durability and reducing the need for clients to resend operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored in volatile system memory for fast access, then speed of data access is improved, but durability of data is worsened due to power failures or reboots
Solution Approach 1:
The patent implements a log object storage mechanism that preemptively records file system operations before they are executed on file objects in volatile memory. This preliminary logging ensures that even if power fails or reboot occurs, the operations can be replayed to restore data, thus resolving the contradiction between fast volatile storage and data durability.
2Reliability
If file system operations are logged and replayed to restore data after power failure, then durability of data is improved, but device complexity is worsened
Solution Approach 1:
The patent introduces log objects as an intermediary layer between file system operations and the actual file objects in volatile memory. These log objects serve as a mediator that records operations and enables recovery without requiring complex changes to the underlying storage hardware or file system structure, thus managing complexity while improving durability.
3Reliability
If clients resend file system operations after data loss, then data integrity is maintained, but network bandwidth usage and resource consumption increase
Solution Approach 1:
The patent enables the storage system to self-restore file objects by replaying logged operations after power failures or reboots. This self-service capability eliminates the need for clients to detect data loss and resend operations, thereby maintaining data integrity while avoiding the additional network bandwidth consumption and resource usage that would result from client-initiated resending.
Data Source
AI summary
A file system interface receives, from one or more clients, file system operations over a network. In response to receiving a file system operation for a file, a file-object manager calls an object storage interface to store, via an application-level network protocol, a log object indicating the file system operation into a file system log in the object storage system and creates or updates a file object in a file object construction buffer in volatile system memory. To evict the file object, the file-object manager calls the object storage interface to store the file object into a file object store in the object storage system. The file-object manager does not have access to local persistent storage. After power failure, the file-object manager calls the object storage interface to retrieve log objects from the file system log and replays file system operations to restore file objects in the volatile system memory.