Chunked Metadata Sync via Single Streaming Request
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current delta sync technology requires many sequential roundtrips between a server and a client or a large payload size, which is resource-intensive and fails in scenarios with limited CPU resources or spotty network connections, leading to inefficiencies and potential failures in syncing metadata changes for large hierarchical datasets.
Innovation Solution
Implementing a chunked response system where a single streaming request is sent for updated metadata, processed in pages, allowing for efficient syncing and resuming metadata updates even after network errors, reducing compute resources and network bandwidth usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If many sequential roundtrips are used for delta sync, then metadata can be synced completely, but processor time and network resources are excessively consumed
Solution Approach 1:
The metadata sync response is segmented into multiple pages, where each page contains a portion of the metadata. The client can process pages sequentially as they are received, avoiding the need to wait for all metadata to be transferred before processing begins. This segmentation allows the system to make progress on processing while network transfer is still in progress, reducing overall processor idle time and enabling better resource utilization.
2Productivity
If one large payload size is used for metadata transfer, then network roundtrips are reduced, but network reliability deteriorates in spotty network conditions
Solution Approach 1:
The large metadata payload is divided into smaller page-sized chunks. Each page can be independently transferred and processed. If a network error occurs, only the incomplete pages need to be retransmitted rather than the entire large payload. This segmentation improves network reliability in spotty conditions while maintaining reasonable network efficiency through reduced roundtrip overhead compared to multiple small requests.
3Reliability
If many sequential roundtrips are used, then complete metadata sync is achieved, but network bandwidth is excessively consumed
Solution Approach 1:
The server prepares and sends multiple pages of metadata in advance within a single network response. The client can begin processing earlier pages while later pages are still being transferred, eliminating the need for multiple sequential roundtrips to retrieve different portions of the metadata. This preliminary action of sending all needed pages in one go reduces network bandwidth consumption while ensuring complete metadata synchronization.
Data Source
AI summary
Aspects of the present disclosure relate to systems and methods for optimizing syncing of metadata changes using a chunked response. In one aspect, an application may be launched in a foreground of a client computing device. A single streaming request may be sent to an application service for updated metadata of a dataset while the application is running in the foreground. A response to the single streaming request may be received at the client computing device. The response may include the updated metadata of the dataset, where the updated metadata is represented as a plurality of pages. While the application is running in a background of the client computing device, each page of the plurality of pages may be processed as the response to the single streaming request is received.