File Duplication System with Parallel Hash and Upload Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing file duplication mechanisms in cloud storage services are inefficient for large files, as calculating unique file identifiers on user devices can be time and resource-consuming, especially on devices with low processing capabilities.
Innovation Solution
A method that estimates the time required to generate a unique file identifier and upload a file to the cloud storage server, initiating uploading if the identifier generation time exceeds upload time, and vice versa, while considering bandwidth, computational resources, and file characteristics, with smart switching between client-side and server-side hash calculations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If unique file identifier is calculated on user device before upload, then server can identify duplicate files and avoid downloading them again, but for large files this calculation becomes time and resource consuming
Solution Approach 1:
The system performs preliminary actions by starting file upload and identifier generation simultaneously rather than sequentially. The upload process begins first, and identifier generation starts in parallel, allowing both operations to progress concurrently and reducing total time required.
Solution Approach 2:
The system skips the traditional sequential workflow where identifier generation must complete before upload starts. By allowing overlapping execution and early termination of identifier generation if upload completes first, the system rushes through the process more efficiently.
2Productivity
If unique file identifier is calculated on user device, then computational resources at user device are utilized, but for devices with low processing capabilities this becomes inefficient
Solution Approach 1:
The system dynamically adjusts where identifier generation occurs based on real-time conditions. It can switch between client-side generation (when resources allow) and server-side generation (when resources are constrained), making the system adaptable to different device capabilities and network conditions.
Solution Approach 2:
The system changes the parameter of where computation occurs (client vs. server) based on file size, network bandwidth, and device capabilities. This parameter change allows optimization of resource usage across different scenarios.
3Loss of time
If file upload is initiated without prior identifier calculation, then upload time is reduced, but server cannot identify duplicates and must download and process entire file
Solution Approach 1:
The system performs preliminary identifier generation in parallel with upload initiation, so that by the time the file reaches the server, the identifier is already available or nearly ready, allowing immediate duplicate checking without waiting for complete file reception.
Solution Approach 2:
The system maintains continuous useful action by overlapping identifier generation with file upload. Both processes run simultaneously rather than one waiting for the other, maximizing resource utilization and reducing idle time.
Data Source
AI summary
Disclosed are systems, methods and computer program products for duplicating files on client device for storage in a cloud storage server. An example method includes: receiving, by a user device, a request to upload a file stored locally on the user device into the cloud storage server; determine time required to generate a unique file identifier for said file and time required to upload said file to the cloud storage server; responsive to the time required to generate the unique file identifier for said file exceeding the time required to upload said file to the cloud storage server, initiating the uploading of the file to the cloud storage server; and responsive to the time required to upload said file to the cloud storage server exceeding the time required to generate the unique file identifier, generating said unique file identifier and transmitting the unique file identifier to the cloud storage server.