Cloud Storage Resume Support for File Transfer Failures
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current cloud-based storage solutions face inefficiencies due to the need to restart file transfers from the beginning after failures, leading to increased backup time and cost, as partially written files must be resent in their entirety.
Innovation Solution
A method that stores file offsets and resumes file transfers from the last successfully written point, using a cloud storage operation platform with a resume database to manage file transfer operations and avoid re-sending data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If file transfer restarts from the beginning after failure, then data integrity is ensured, but backup time and cost increase significantly
Solution Approach 1:
The system performs preliminary actions by storing metadata including file offsets and transfer status information in a database before failures occur. This allows the system to quickly resume from the last successful offset without retransmitting data, while still ensuring integrity through verification mechanisms.
Solution Approach 2:
The system implements feedback by continuously updating and storing transfer status information in a database during the file transfer process. After failure, this feedback information is used to determine the resumption point, allowing the system to recover efficiently while maintaining data integrity through verification.
2Reliability
If file transfer restarts from the beginning after failure, then data consistency is maintained, but network traffic and cost increase
Solution Approach 1:
The system stores metadata including current file offsets and transfer status in a database during normal operation. This preliminary action enables resumption from the exact offset where transfer left off, avoiding redundant network traffic while maintaining consistency through verification protocols.
Solution Approach 2:
The system uses feedback from stored transfer status information to intelligently determine where to resume transmission. This feedback mechanism ensures that only the necessary portion of data is retransmitted, reducing network traffic while maintaining data consistency through offset tracking.
3Reliability
If complete files are retransmitted after partial write failure, then cloud storage integrity is ensured, but operational efficiency decreases
Solution Approach 1:
The system segments the file transfer process into manageable chunks with tracked offsets. By storing metadata about completed segments in a database, the system can resume only the uncompleted segments after failure, maintaining cloud storage integrity while dramatically improving operational efficiency.
Solution Approach 2:
The system performs preliminary tracking of transfer progress by storing offset information in a database during normal operation. This allows efficient resumption of only the necessary file segments after failure, rather than retransmitting entire files, thus improving productivity while ensuring integrity.
Data Source
AI summary
A method comprises storing information corresponding to a plurality of files being written from a source storage device to a cloud storage platform in a file transfer operation. The method further comprises, in response to a failure of the file transfer operation, identifying from the information one or more file offsets successfully written to the cloud storage platform for one or more of the plurality of the files prior to the failure. The file transfer operation for the one or more of the plurality of files is resumed from the identified one or more file offsets.