Proxy Server Chunk Upload for Cloud File Reliability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing file upload methods in cloud services are inefficient, particularly for large files, as they require numerous requests and transactions over slow networks, leading to prolonged upload times and increased load on backend servers, with errors necessitating re-upload of entire files and delaying subsequent file uploads.
Innovation Solution
A proxy server is introduced within a cloud computing environment that divides files into chunks, generates upload links, and simultaneously uploads these chunks to cloud storage, re-uploading only faulty chunks and extracting metadata for post-processing, thereby minimizing network transactions and enabling faster upload speeds and error recovery.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If files are uploaded by dividing into chunks and generating multiple requests, then upload reliability is improved through error recovery, but upload time increases due to numerous network transactions
Solution Approach 1:
A proxy server is introduced as an intermediary between the client and cloud storage. The proxy server receives the entire file from the client, divides it into chunks, generates upload links, and uploads chunks to cloud storage. This mediator approach allows the client to send fewer requests while maintaining the chunk-based reliable upload mechanism, thus reducing upload time without sacrificing reliability.
Solution Approach 2:
The proxy server performs preliminary actions by pre-generating upload links for all chunks before the actual upload process. This allows the client to upload files continuously without waiting for link generation, reducing idle time and overall upload duration while maintaining the ability to recover from errors through the chunk-based approach.
2Reliability
If multiple requests are generated for file upload, then upload completeness is improved through error detection, but network transaction overhead increases
Solution Approach 1:
The proxy server acts as an intermediary that consolidates multiple upload requests into a single client-to-proxy transaction. The proxy server then handles the complexity of generating multiple upload links and uploading individual chunks to cloud storage, thereby reducing network transaction overhead between client and cloud while maintaining upload completeness through error detection mechanisms.
Solution Approach 2:
The upload process is segmented into distinct functional components: file reception, chunk division, link generation, and chunk upload. Each segment is handled by the proxy server, which manages the complexity internally. This segmentation allows the system to maintain comprehensive error detection through multiple operations while presenting a simplified interface to the client, reducing perceived network transaction overhead.
3Reliability
If entire files are re-uploaded upon error occurrence, then upload reliability is maintained, but time loss increases significantly
Solution Approach 1:
The file is segmented into multiple independent chunks, each with its own upload link. When an error occurs during upload, only the specific failed chunk needs to be re-uploaded rather than the entire file. This segmentation maintains upload integrity through error detection while significantly reducing time loss by limiting re-upload scope to only the affected segment.
Solution Approach 2:
Upload links for all chunks are generated in advance before the upload process begins. This preliminary generation of links allows for quick identification and re-upload of only the specific chunk that failed, rather than regenerating all links or re-uploading the entire file. This approach maintains upload integrity while minimizing time loss through targeted error recovery.
Data Source
AI summary
A proxy server for uploading a file according to an embodiment of the present disclosure improves the upload speed of a file and to minimize upload delay due to the occurrence of an error in the upload process. Due to the fact that the upload of the upload target file is started by the client delivering only one HTTP request to the proxy server to start the upload of the upload target file instead of requesting an upload link for each divided chunk to the backend server and receiving the response message from the backend server, a transaction over a network having a relatively slow communication speed can be minimized and the upload speed can be improved accordingly.