Cloud Upload Partial Read Access During Multipart Transfer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional cloud storage systems require users and downstream consumers to wait until the entirety of a content upload is complete before accessing it, causing delays and bottlenecks in processing large files.
Innovation Solution
Implement a mechanism that allows access to uploaded content before its completion by using a read module to track and provide partial data during multi-part uploads, enabling active or reactive shadow structure tracking of uploaded parts and byte ranges.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the system waits for complete content upload before providing access, then data integrity is ensured, but downstream processing is delayed
Solution Approach 1:
The system performs preliminary actions by creating a placeholder object and shadow structure before the complete upload finishes. This allows downstream consumers to access and process available data portions early, while the upload continues in the background. The placeholder object is prepared in advance with metadata, enabling immediate access to already-uploaded portions without waiting for completeness.
Solution Approach 2:
The upload process is segmented into multiple independent parts that can be uploaded, accessed, and processed separately. Each part of the content can be independently accessed by downstream consumers once uploaded, without requiring the entire content to be complete. This segmentation allows partial data access while maintaining overall data integrity through the shadow structure tracking.
2Productivity
If the system allows partial data access during upload, then downstream workflow efficiency is improved, but system complexity increases
Solution Approach 1:
The shadow structure acts as an intermediary between the upload process and downstream consumers. It tracks which parts of the content have been uploaded and are accessible, providing a simplified interface for consumers to query and access available data without needing to understand the complex multipart upload state. This intermediary layer manages the complexity internally while presenting a simple access model externally.
Solution Approach 2:
The system creates a shadow copy or placeholder object that mirrors the structure of the final content object. This shadow structure copies the metadata and organization of the expected final object, allowing downstream consumers to interact with a simplified representation rather than the complex ongoing upload process. The shadow structure is updated as parts are uploaded, providing an accurate but simplified view of available data.
3Ease of operation
If the system tracks uploaded parts using shadow structure, then partial access is enabled, but computational overhead increases
Solution Approach 1:
The shadow structure tracks only the portions of data that have been uploaded and are accessible, rather than maintaining complete information about the entire content object. This partial tracking approach enables partial data access functionality without the computational cost of tracking every possible byte or part. The system performs just enough tracking to enable the required partial access capability.
4Productivity
If the system provides early access to uploaded content, then downstream processing bottlenecks are reduced, but data consistency challenges arise
Solution Approach 1:
The shadow structure provides continuous feedback about the state of uploaded parts to downstream consumers. As parts are uploaded, the shadow structure is updated and this state information is fed back to consumers, allowing them to know exactly which portions of data are currently accessible. This feedback mechanism ensures data consistency by preventing consumers from accessing incomplete or incorrect data portions while still enabling early access to available data.
Data Source
AI summary
Disclosed is an improved approach to provide access to uploaded content even before the entirety of the content has been uploaded to cloud-based storage. This permits the user and/or the user's authorized services or data consumers to read partial data while multi-part upload[s] are still in progress.