Asynchronous Object Upload in Content-Addressable Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems face delays in uploading large objects due to the need for splitting them into chunks and uploading each chunk individually, which can be exacerbated by client and server failures, and there is a need for efficient distributed object uploading that provides consistency guarantees.
Innovation Solution
A method for asynchronous distributed object uploading in a streaming mode using a distributed multi-master data replication system, where objects are split into chunks, uploaded to available clusters, and once finalized, a unique content handle is computed and the object is inserted into a content-addressable index as an immutable object, ensuring availability, liveness, and consistency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If objects are split into chunks and uploaded individually in streaming mode, then upload capability for large objects is enabled, but upload time increases significantly
Solution Approach 1:
The patent applies segmentation by dividing large objects into chunks that can be uploaded individually in streaming mode. This allows the system to handle large objects that would otherwise be too large to upload efficiently, while the chunked approach enables parallel processing and reduces overall upload time compared to traditional monolithic uploads.
2Productivity
If streaming mode upload is used for large objects, then upload efficiency improves, but consistency guarantees become more difficult to maintain
Solution Approach 1:
The patent implements feedback mechanisms through acknowledgment tracking and finalization processes. The system tracks which chunks have been successfully uploaded and received acknowledgments, and provides feedback to clients about upload status. This feedback loop enables the system to maintain consistency guarantees by ensuring all chunks are properly accounted for before considering the upload complete, while still allowing efficient streaming mode operation.
3Adaptability or versatility
If multi-master model is used for dynamic object creation across clusters, then system scalability improves, but write-order fidelity becomes more complex to maintain
Solution Approach 1:
The patent introduces an intermediary replication layer that sits between the multi-master object creation mechanism and the actual data storage. This intermediary layer automatically maintains write-order fidelity by ensuring that replicated objects are created in the correct order across all clusters, even when using the scalable multi-master model. The intermediary handles the complexity of coordinating write operations across distributed clusters while allowing the system to maintain scalability.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method is performed by two or more devices of a group of devices in a distributed data replication system. The method includes receiving, at the two or more devices, a group of chunks having a same unique temporary identifier, where the group of chunks comprises an object to be uploaded; creating an entry for the object in a replicated index, where the entry is keyed by the unique temporary identifier, and where the replicated index is replicated at each of the two or more devices; and determining, by an initiating device of the two or more devices, that a union of the group of chunks contains all data of the object. The method also includes calculating a content-based identifier to the object; creating another entry for the object in the replicated index, where the other entry is keyed by the content-based identifier; and updating the replicated index to point from the unique temporary identifier to the content-based identifier.