Crowd-Sourced Cache for Data Restore Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing backup and restore systems using object storage platforms face slow restore times due to high latency and inefficiencies in utilizing the capabilities of these systems, particularly because they rely on dedicated backup infrastructure and do not fully leverage the parallelism and scale of object storage platforms.
Innovation Solution
A system and method that utilizes a dynamic crowd-sourced cache of objects across a client population, assessing object references to create a distributed cache based on available storage, network, and computing resources, prioritizing local resources for data restore requests, and bypassing the backup server bottleneck by using multithreaded processes and peer client resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If traditional dedicated backup infrastructure is used to retrieve objects from object storage platforms, then data restore can be performed, but restore times are slow due to high latency and the need to retrieve entire objects even when only changed data is needed
Solution Approach 1:
The patent segments the data retrieval process by identifying and extracting only the specific changed chunks that need to be restored, rather than retrieving entire objects. The system divides objects into chunks, tracks which chunks have changed, and selectively retrieves only those specific chunks from the object storage platform, significantly reducing restore time and data transfer requirements
Solution Approach 2:
The system performs preliminary actions by pre-processing data during backup operations to identify changed chunks and create a deduplication database that maps object references to specific chunk locations. This preliminary chunk-level tracking and indexing enables efficient selective retrieval during restore operations, avoiding the need to fetch entire objects
2Quantity of substance
If data deduplication is performed to reduce storage costs by storing only changed data, then storage efficiency improves, but restore operations become slower because traditional systems must retrieve entire objects despite only needing changed portions
Solution Approach 1:
The patent segments objects into manageable chunks and maintains a deduplication database that tracks chunk-level changes and references. This segmentation enables the system to store only changed chunks while maintaining efficient indexes that allow rapid identification and retrieval of specific chunks during restore operations, eliminating the need to retrieve entire objects
Solution Approach 2:
The system introduces a deduplication database as an intermediary layer between the object storage platform and restore operations. This database stores metadata about chunk locations, versions, and references, enabling the system to quickly identify which specific chunks need to be retrieved without scanning or downloading entire objects from storage
3Quantity of substance
If object storage platforms are used for massive scale and parallelism, then storage capacity and scalability improve, but latency increases compared to other storage types
Solution Approach 1:
By segmenting data into chunks and enabling selective retrieval of specific chunks rather than entire objects, the system reduces the amount of data that needs to be transferred over the network, effectively mitigating the latency inherent in object storage platforms while maintaining the benefits of massive scale and parallelism
Solution Approach 2:
The system performs partial action by retrieving only the specific chunks that have changed and are needed for restore operations, rather than performing excessive action by retrieving entire objects. This partial retrieval approach significantly reduces data transfer volume and network latency while still achieving complete restore functionality
Data Source
AI summary
A computer-implemented method includes: receiving, by a computing device, a data restore request; determining, by the computing device, at least one object in response to the data restore request; and querying, by the computing device, a client population to determine at least one client which has resources and network cardinality to assist the data restore request. The at least one client creates a distributed and crowd sourced cache of objects for the data restore request.