Consistent User View System for Partitioned Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale services face issues with eventual consistency due to propagation delays in data replication across partitions, leading to user dissatisfaction when changes are not immediately visible after confirmation, especially when load balancing becomes a scalability bottleneck during outages.
Innovation Solution
A consistent user view system is implemented with separate layers for handling write and read requests, using a journal for write requests and a store for read requests, along with a user cache for storing user-specific changes, ensuring that users see their changes reflected even before they are fully propagated to all partitions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is replicated across multiple partitions to allow different views and increase scalability, then the service can handle larger volumes of data, but propagation delays cause users to see inconsistent data after making changes
Solution Approach 1:
The system segments data storage into separate journal partitions and store partitions, allowing independent management of write operations and read operations. This segmentation enables the journal to accept writes immediately while the store gradually propagates changes, resolving the consistency issue while maintaining scalability.
Solution Approach 2:
The journal acts as an intermediary between user write operations and the store. It buffers write requests and manages their propagation to multiple store partitions, ensuring that users see their changes immediately through the journal while maintaining eventual consistency across all partitions.
2Loss of information
If load balancing is used to direct users to the same partition, then data consistency is maintained, but the system becomes a scalability bottleneck during outages
Solution Approach 1:
The system separates write handling (journal) from read handling (store), allowing reads to be distributed across multiple partitions without affecting write consistency. This eliminates the need for load balancing to direct users to a single partition, improving scalability while maintaining consistency through the journal's write-ahead logging mechanism.
3Loss of information
If synchronization mechanisms are used to reconcile data across partitions, then eventual consistency is achieved, but changes take time to replicate causing user confusion
Solution Approach 1:
The journal performs preliminary action by recording all write operations before they are propagated to the store. This write-ahead approach ensures that users can immediately see their changes in the journal, eliminating propagation delay perception while the store gradually synchronizes in the background.
Data Source
AI summary
A consistent user view system. The system incorporates any changes made by a user in any views shown to that user even when the changes have not propagated to the partitions supplying the view. The system separates the authority for edits from the replicated storage allowing efficient transactions and linear scalability. Documents are read from view-based partitions of a store. Document writes are written to a document-specific partition in a journal and applied to the store. The system stores a copy of pending changes in a user-specific partition. When a user requests a view, the system checks that user's cache for any pending changes applicable to the view. If any applicable changes are found, the changes are applied before showing the view to the user. Pending changes that have been successfully applied to the store are trimmed from the user-specific partition to free up resources.