Stream-Based Logging for Distributed Storage Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current elastic computing systems face challenges in managing data replicas, leading to high latencies due to bandwidth limitations and difficulties in balancing bandwidth and latency across distributed systems, particularly in handling user I/O operations and ensuring data resiliency.
Innovation Solution
The implementation of a highly distributed tertiary replica system, where data is split into a large number of partitions across multiple devices, allowing for asynchronous updating and utilizing a logger fleet to manage message streams for efficient data replication and recovery, while maintaining primary and secondary replicas for real-time user interactions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is replicated across multiple devices in a distributed system, then data resiliency is improved, but latency increases due to bandwidth limitations
Solution Approach 1:
The patent divides data into multiple partitions that are distributed across different devices. Each partition can be independently managed and replicated, allowing the system to maintain data resiliency while reducing the latency impact on individual data access operations. The segmentation enables parallel processing and reduces contention for bandwidth resources.
Solution Approach 2:
The patent introduces a logger fleet as an intermediary component that manages message streams between data producers and consumers. This intermediary layer optimizes data flow and replication processes, reducing the latency impact of distributed replication while maintaining data resiliency through coordinated message delivery.
2Reliability
If bandwidth is allocated for data replication, then data resiliency is improved, but user I/O performance deteriorates
Solution Approach 1:
The patent implements dynamic bandwidth allocation where the system can adjust replication bandwidth based on current system conditions and user I/O demands. During periods of high user I/O activity, replication bandwidth is reduced to prioritize user performance, while during low-utilization periods, replication bandwidth is increased to maintain data resiliency.
Solution Approach 2:
The patent performs preliminary data replication and buffering actions in advance of user I/O operations. By proactively replicating data partitions and maintaining message streams before user requests arrive, the system reduces the need for real-time replication during user I/O operations, thereby minimizing the impact on user performance while maintaining data resiliency.
3Reliability
If synchronous replication is used, then data consistency is improved, but system throughput deteriorates
Solution Approach 1:
The patent implements periodic consistency checks and asynchronous replication with periodic synchronization. Instead of requiring synchronous acknowledgment for every write operation, the system replicates data asynchronously and performs periodic consistency verification. This approach maintains data consistency over time while significantly improving system throughput by eliminating synchronous blocking.
Solution Approach 2:
The patent creates multiple copies of data partitions that can be independently updated and managed. Rather than requiring all replicas to be synchronized for every operation, the system maintains multiple independent copies that can be reconciled periodically, improving throughput while maintaining overall data consistency through the copying mechanism.
Data Source
AI summary
Generally described, aspects of the present application correspond to maintaining a message stream for a network-based data store, which stream includes messages reflecting modifications to the data store. Messages within the stream may be used to revert a state of the data store to a prior point in time reflected within the messages of the stream, such as by “rewinding” operations on the data store by use of the messages within the stream. Messages in the stream may further be used to asynchronously update a replica of the data store.