Aggregated Small File Snapshot Compatibility in Distributed Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed file storage systems like CephFS, small files aggregated into large files lead to inefficient resource utilization and snapshot functionality failures due to the Copy-On-Write mechanism not being triggered correctly during write and deletion operations.
Innovation Solution
A method and apparatus that involves a Metadata Server (MDS) determining the presence of an O_TRUNC identifier in a file, converting aggregated small files into normal small files when necessary, and triggering Copy-On-Write operations to ensure snapshot data integrity during opening, writing, and deletion operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If small files are aggregated into large files to improve resource utilization, then storage efficiency is improved, but snapshot functionality fails because COW cannot be triggered correctly
Solution Approach 1:
The system performs preliminary detection when a file is opened to check if it is an aggregated small file with snapshots. If detected, the system converts the aggregated file to a normal file structure before allowing write operations, ensuring COW can be triggered correctly in the future. This preliminary action prevents the snapshot functionality failure before it occurs.
Solution Approach 2:
The system introduces an intermediary conversion process that transforms aggregated small files into normal file structures when snapshots are detected. This intermediary step (conversion to normal file) acts as a mediator between the aggregated file state and the required normal state for COW operation, resolving the contradiction between storage efficiency and snapshot functionality.
2Ease of operation
If aggregated small files are opened with O_TRUNC to enable write operations, then write capability is restored, but snapshot data integrity is compromised
Solution Approach 1:
Before allowing O_TRUNC write operations on aggregated small files, the system performs preliminary detection and conversion to normal file structure. This ensures that when write operations occur, COW can be triggered correctly and snapshot data integrity is maintained, preventing information loss while enabling write capability.
Solution Approach 2:
The system implements feedback through detection and conversion logic that monitors the file state. When an aggregated small file with snapshots is detected, the system converts it to a normal file structure, providing feedback that prevents incorrect COW triggering and protects snapshot data integrity during subsequent write operations.
Data Source
AI summary
A method and apparatus for compatibility between a snapshot and small file aggregation under a distributed file storage system, and a computer device and a storage medium are provided. The method includes: receiving, by a Metadata Server (MDS), an operation request sent by a client, and when the operation request is an opening request, determining whether there is an O_TRUNC identifier in a file; when there is the O_TRUNC identifier in the file, continuing to determine whether the file is an aggregated small file and has a snapshot; when the file is the aggregated small file and has the snapshot, returning an error indicator to the client; and converting, by the client, the aggregated small file into a normal small file after receiving the error indicator; and triggering a Copy-On-Write (COW) or deletion operation of an object of the small file, so as to ensure that snapshot data is correct.