Metadata-Linked Stub Objects for Multipart Upload Completion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The process of completing a multi-part upload (MPU) in cloud storage systems is time-consuming due to the need for spillover reads and writes, which are proportional to the size of the combined parts, leading to inefficiencies in handling large data objects.
Innovation Solution
Generating a stub object with metadata that references the uploaded parts of a multipart upload object, avoiding spillover by creating a permanent index structure and using metadata to access the parts, thus reducing the need for disk reads and writes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the complete command is executed to finalize the multipart upload object, then the stored data object is generated from uploaded parts, but the completion time becomes prohibitively time consuming due to spillover reads and writes proportional to the size of the combined parts
Solution Approach 1:
The patent segments the multipart upload completion process into two distinct phases: (1) uploading parts individually which can be done in parallel without coordination overhead, and (2) a finalization step that uses a stub object with metadata to reference the uploaded parts without requiring spillover reads and writes of the entire combined data. This segmentation eliminates the time-consuming spillover operations while maintaining data integrity.
Solution Approach 2:
The patent creates a stub object that contains metadata referencing the uploaded parts instead of physically consolidating all parts into a single object. The stub object acts as a lightweight copy or representation of the final multipart upload object, allowing the system to track and manage the uploaded parts without performing expensive spillover reads and writes to create the actual consolidated object.
2Reliability
If spillover reads and writes are performed to create the multipart upload object, then the object is properly finalized, but the processing overhead increases proportionally to the size of the combined parts
Solution Approach 1:
The patent extracts the essential metadata information (part keys, sizes, offsets) from the uploaded parts and stores it in a stub object. This extraction allows the system to maintain object integrity and track uploaded parts without performing the expensive operation of reading and writing the actual part data during completion. The stub object contains only the necessary references to reconstruct or access the full object when needed.
Solution Approach 2:
The stub object serves as an intermediary between the uploaded parts and the final multipart upload object. Instead of directly consolidating all parts through spillover reads and writes, the stub object mediates by storing metadata that references the parts, allowing the system to verify completion and enable future access without the expensive intermediate consolidation step.
3Ease of operation
If the multipart upload object is stored with all parts consolidated, then random access to any part is simplified, but the storage space and access time increase due to the need to handle the entire combined object
Solution Approach 1:
The patent maintains the segmented structure of uploaded parts while introducing a stub object that contains metadata (offsets, sizes, keys) for each part. This allows the system to access specific parts directly using the metadata information without reading or writing the entire consolidated object, achieving both ease of access and time efficiency.
Solution Approach 2:
The patent adds a metadata dimension through the stub object that provides indirect access to the uploaded parts. Instead of accessing parts through their physical location in a consolidated object, the system uses the metadata dimension (stored in the stub object) to quickly locate and access specific parts, effectively adding a new dimension of access that avoids the time cost of handling the entire combined object.
Data Source
AI summary
The technologies described herein are generally directed toward generating a stub object with metadata referencing uploaded parts of a multipart upload object. In an embodiment, a method can include receiving a multipart upload object comprising references to upload parts. Further, the method can include, based on the multipart upload object being predicted to be subject to spillover during creation of the multipart upload object according to the upload parts, identifying the upload parts referenced by the multipart upload object, resulting in identified upload parts. The method can further include generating a stub object with metadata referencing the identified upload parts of the multipart upload object.