Edge Server File Pre-Replication for Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing file sharing systems experience significant latency during the initial download of shared files, as edge servers often need to interface with distant servers in the storage system, leading to undesirable download times for users.
Innovation Solution
A system and method that forecasts likely edge servers for file downloads and pre-replicates files onto these servers before a download request is made, using a prediction model based on historical download data to identify high-likelihood edge servers and replicate files accordingly, thereby reducing latency and improving download speeds.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If files are stored only in the distributed storage system without pre-replication to edge servers, then storage system complexity is reduced, but download latency increases significantly
Solution Approach 1:
The system performs preliminary replication of files to edge servers based on predicted download locations and patterns. By proactively copying files to edge servers before actual download requests occur, the system reduces download latency without requiring permanent storage at edge locations, thus managing complexity through temporary caching rather than permanent distribution
Solution Approach 2:
Edge servers act as intermediaries between the distributed storage system and endpoint devices. These intermediaries cache files temporarily and serve download requests locally, reducing the need for direct communication between endpoint devices and distant storage servers, thereby reducing latency while maintaining a relatively simple core storage architecture
2Speed
If files are pre-replicated to all edge servers, then download speed is improved, but network bandwidth consumption and storage costs increase
Solution Approach 1:
Instead of uniform replication to all edge servers, the system applies local quality by replicating files only to specific edge servers based on predicted download locations, user patterns, and geographic distribution. This targeted approach ensures fast downloads for likely users while minimizing unnecessary network bandwidth consumption and storage usage at edge servers that are unlikely to be accessed
Solution Approach 2:
The system performs partial replication by copying only the portions of files that are most likely to be downloaded, or replicating to a subset of edge servers rather than all of them. This partial action reduces network bandwidth consumption and storage costs while still achieving acceptable download speeds for the majority of use cases
3Loss of time
If the system uses a prediction model to forecast edge server usage, then download latency is reduced, but computational overhead and system complexity increase
Solution Approach 1:
The prediction model uses feedback from historical download data, user behavior patterns, and system performance metrics to continuously improve its forecasts of which edge servers will be used. This feedback mechanism allows the system to refine its predictions over time, reducing computational overhead by focusing resources on high-probability scenarios while maintaining low download latency through accurate forecasting
Data Source
AI summary
A system and method of providing early replication of files to edge devices during a file share operation. A system is provided that is configured to receive a file on behalf of a network of devices, the network of devices including the computing device and is configured to store the file. The device is further configured to identify an edge device based on previous file downloads for a given pair of users, the edge device configured to enable download of a copy of the file to an endpoint device; and provide the copy of the file to the edge device before receipt of a request by the edge device from the endpoint device to download the copy of the file.