Geographic Object Storage Fast Recall via Local Data Caching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Geographically distributed erasure coding object storage systems face challenges in retrieving data quickly due to the need to reconstruct object data from distributed chunks across multiple storage locations, which can be slow and costly, especially over wide-area networks.
Innovation Solution
Implement a distributed topology that maintains a complete instance of object data at a storage location for a threshold time based on access patterns, using last access data structures and time thresholds to anticipate and optimize data access, while reducing storage redundancy and improving retrieval times.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored using erasure coding across geographically distributed locations, then data protection and redundancy are improved, but data retrieval speed and cost are worsened
Solution Approach 1:
The system performs preliminary action by maintaining a complete instance of object data at the first storage location before any retrieval request occurs. This complete instance is kept for a threshold time period based on access patterns, so that when a retrieval request comes in, the data can be returned immediately without needing to reconstruct it from distributed chunks across multiple geographic locations.
2Loss of time
If complete instance of object data is maintained at storage location, then data retrieval time is improved, but storage cost and redundancy are worsened
Solution Approach 1:
The system applies dynamics by making the storage strategy adaptive rather than static. It monitors access patterns and dynamically adjusts whether to maintain a complete instance of object data or revert to chunked storage. The time threshold parameter allows the system to automatically transition between these states based on whether the current time exceeds the threshold since last access, enabling flexible optimization between retrieval speed and storage cost.
3Reliability
If data chunks are distributed across multiple systems, then storage redundancy and protection are improved, but device complexity and coordination overhead are worsened
Solution Approach 1:
The system extracts the complexity of data coordination and reconstruction by maintaining a complete instance of object data locally at the first storage location. This eliminates the need for complex coordination between multiple geographically distributed systems for data retrieval operations, as the complete data can be served from a single location without needing to orchestrate chunk assembly across the network.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
In some examples, a computer system may receive a data object including object data. For example, the system may be one of three or more systems located at different respective geographic locations. The system may determine a plurality of chunks by dividing the object data into a plurality of data chunks based on a total number of the systems and determining a parity chunk. The system may send some of the chunks to the other systems, while maintaining a complete instance of the object data at the first system. The system may associate a time with the object data based on receiving the data object. Additionally, the system may associate a time threshold with the complete instance of the object data. For example, the time threshold may indicate a time at which the complete instance of the object data is to be deleted from the system.