Cloud Controller File Deletion in Distributed Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing network-based storage systems face challenges in maintaining data consistency and performance, especially in cloud-based storage solutions, where access latency is increased, and network failures can lead to data unavailability and consistency issues.
Innovation Solution
A distributed filesystem managed by cloud controllers that cache and ensure data consistency across multiple cloud storage systems, using incremental metadata snapshots and deduplication techniques to maintain data integrity and visibility, while performing background deletion operations to avoid performance impact on other users.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If cloud-based storage systems are used to provide large-scale remote network storage solutions, then storage capacity and ease of expansion are improved, but access latency increases and data consistency problems occur
Solution Approach 1:
The system segments storage functionality into cloud controllers that maintain local caches of frequently accessed data. This segmentation allows the system to provide large-scale storage capacity through cloud storage systems while maintaining fast access latency by serving frequently accessed data from local caches, thus resolving the contradiction between storage capacity and access speed.
2Ease of operation
If cloud-based storage systems are used to simplify storage management, then management overhead is reduced, but network failures can prevent data access for significant periods
Solution Approach 1:
The system implements prior cushioning by maintaining local caches of data and metadata in cloud controllers before network failures occur. This preparatory caching ensures that frequently accessed data remains available even when network connections to cloud storage systems fail, thus maintaining data availability and system reliability while keeping management overhead low.
3Reliability
If file deletion operations are performed synchronously in cloud storage systems, then data consistency is maintained, but performance of other users is negatively affected
Solution Approach 1:
The system performs preliminary action by updating the user view of the filesystem to present the appearance of file deletion immediately, while initiating background deletion operations separately. This allows data consistency to be maintained from the user perspective while avoiding performance degradation for other users, as the actual deletion of data blocks occurs asynchronously in the background without blocking other operations.
Data Source
AI summary
The disclosed embodiments disclose techniques for deleting a file from a distributed filesystem. Two or more cloud controllers collectively manage distributed filesystem data that is stored in one or more cloud storage systems; the cloud controllers store metadata for the distributed filesystem, and cache and ensure data consistency for the data stored in the cloud storage systems. During operation, a cloud controller receives a request from a client to delete a file from the distributed filesystem. The cloud controller updates a user view of the distributed filesystem to present the appearance of the target file being deleted to the client, and then initiates a background deletion operation to delete the target file without negatively affecting the performance of the other users of the distributed filesystem.