Distributed Data Replication Using Historical Query Locality
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud computing systems face inefficiencies in data replication due to high latency and bandwidth limitations in cross-server data transfers, leading to scalability issues and storage capacity constraints when replicating data to every server for fault tolerance.
Innovation Solution
A method and system that analyze historical query logs to identify commonly owned data sets and frequently linked projects, using a mixed integer programming model to optimize data storage by replicating only a subset of servers, minimizing remote reads, and ensuring data remains local where possible.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all storage sets are replicated to every server for fault tolerance and high availability, then system reliability is improved, but data transfer costs and storage requirements grow linearly with the number of servers
Solution Approach 1:
The patent applies local quality by making replication decisions specific to each storage set based on its access patterns. Frequently accessed storage sets are replicated to multiple servers, while less frequently accessed sets are replicated to fewer servers. This selective approach optimizes the balance between reliability and data transfer costs by tailoring replication levels to local access requirements rather than applying uniform replication across all data.
Solution Approach 2:
The system dynamically changes the replication parameter (number of copies) based on historical access patterns and predicted future access. By monitoring access frequency and adjusting replication levels accordingly, the system adapts the reliability level to match actual usage, reducing unnecessary data transfer costs while maintaining adequate fault tolerance for important data.
2Reliability
If data is replicated to every server, then high availability is achieved, but storage scalability is limited by the capacity of the smallest server
Solution Approach 1:
Different servers store different subsets of replicated storage sets based on their available capacity and the access patterns of the data. This allows the system to utilize the full storage capacity of each server without being constrained by the smallest server's capacity, while still maintaining high availability through selective replication of critical data to multiple locations.
3Adaptability or versatility
If cross-server reads are allowed beyond a small limit, then data access flexibility is improved, but performance degrades due to higher latency and limited bandwidth
Solution Approach 1:
The system performs preliminary actions by proactively replicating storage sets to servers that are likely to need them, based on historical access patterns and query analysis. By predicting which servers will access which data and pre-positioning the data locally, the system reduces the need for cross-server reads during actual query execution, thereby maintaining data access flexibility while minimizing latency and bandwidth usage.
Data Source
AI summary
Historical analysis of query patterns is used to discover relationships between data sets. These relationships are used to make optimal decisions about where to place data in a globally distributed environment in which locality of data is an important factor in providing good query performance. A mixed integer programming model is used to solve a constraint based system which balances the need to have data kept local with other data and the cost of replicating data across a low-bandwidth network.