Partial Replica Storage System for Data Durability and Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional storage systems in distributed computing environments are limited in their ability to efficiently meet a combined set of storage requirements, including loss of a replica, degraded performance, storage efficiency, and a predefined number of durable copies, often disregarding one or more of these objectives, which affects durability and latency.
Innovation Solution
The implementation of a replica set that includes a partial replica and a set of full replicas, with a change coordinator managing operations to ensure data durability and efficiency by temporarily storing data with the partial replica and forgetting it when full replicas confirm promise responses, allowing for continuous replication operations and in-place reads and writes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If three copies of data are stored to ensure durability, then data loss risk is reduced, but storage resources are overconsumed
Solution Approach 1:
The patent applies local quality by differentiating between full replicas and partial replicas. Full replicas store complete data blocks for durability, while partial replicas store only partial data blocks temporarily. This differentiation allows the system to maintain data durability through full replicas while reducing overall storage resource consumption by using partial replicas for temporary storage during write operations.
Solution Approach 2:
The patent segments the replica set into two distinct types: full replicas and partial replicas. Full replicas maintain complete copies of data blocks for durability guarantees, while partial replicas hold only portions of data blocks temporarily during the write process. This segmentation enables the system to achieve durability with fewer complete copies while using partial copies efficiently for temporary storage.
2Reliability
If a quorum-based storage system with at least three durable copies is used, then data durability is improved, but storage efficiency deteriorates
Solution Approach 1:
The system applies local quality by assigning different roles to different replicas: full replicas provide durability guarantees with complete data blocks, while partial replicas improve storage efficiency by holding only temporary partial data blocks. This allows the system to maintain quorum-based durability while improving overall storage efficiency through the temporary nature of partial replica storage.
Solution Approach 2:
The patent implements discarding and recovering by automatically removing data blocks from partial replicas once the write operation is complete and confirmed by the quorum. This temporary storage approach allows the system to achieve durability through full replicas while improving storage efficiency by discarding temporary partial copies after use, rather than permanently storing them.
3Reliability
If conventional durable storage mechanisms are implemented, then data durability is ensured, but latency increases
Solution Approach 1:
The patent applies preliminary action by having the partial replica temporarily hold data blocks before the write operation is fully committed. This allows the system to prepare for the write operation in advance, reducing latency by having the data already positioned in the partial replica rather than transferring it from the client directly to full replicas during the commit process.
Solution Approach 2:
The partial replica serves as an intermediary between the client and full replicas during write operations. It temporarily holds data blocks and coordinates with the quorum, allowing the system to optimize the write path and reduce latency by processing writes through this intermediate layer rather than directly involving all full replicas in every write operation.
4Productivity
If the replica set includes both partial replica and full replicas, then storage efficiency is improved, but system complexity increases
Solution Approach 1:
The system manages complexity through local quality by giving distinct, well-defined roles to full replicas and partial replicas. Full replicas are responsible for durability with complete data blocks, while partial replicas handle temporary storage of partial data blocks. This clear role differentiation simplifies management compared to treating all replicas uniformly, as each type has specific responsibilities that reduce coordination overhead.
Solution Approach 2:
The patent inverts the conventional approach by using partial replicas rather than requiring all replicas to be full replicas for durability. Instead of having every replica store complete data blocks, the system uses partial replicas for temporary storage and only requires a quorum of full replicas for durability guarantees. This inversion reduces the management complexity of maintaining full copies across all replicas while achieving the same durability goal.
Data Source
AI summary
Methods, systems, and computer storage media for providing temporary storage for data that has to be durably stored based on a replica set that includes a partial replica and a set of full replicas in a distributed computing environment. The partial replica of the replica set temporarily and durably stores data but does not persist the data when the set of full replicas have communicated a promise response to store the data. In operation, instructions to communicate promise responses are communicated to a replica set comprising a partial replica and a set of full replicas. A promise response indicates that a replica in the replica set will commit the data block of the write request. Promise responses are received from the set of full replicas. Based on receiving promise responses from the set of full replicas, a forget instruction is communicated to the partial replica to forget the data block.