Stream-Based Shared Storage System for Distributed Data Consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed database systems face coordination challenges, particularly with lock-based systems that slow operations and become complex as additional nodes are added, and read replica models experience lag due to asynchronous data propagation.
Innovation Solution
A stream-based system that replicates shared data storage among client hosts without using lock-based mechanisms, where client hosts perform read and write operations using stream events stored in a sequence by a stream service, allowing for independent access and asynchronous update propagation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If lock-based systems are implemented to coordinate access to shared data among multiple nodes, then data consistency is maintained, but operation speed decreases and system complexity increases
Solution Approach 1:
The patent extracts the locking mechanism from the data access coordination process entirely. Instead of using locks to coordinate read/write operations, the system uses version vectors and conflict-free replicated data types (CRDTs) that allow concurrent access without mutual exclusion, thereby maintaining data consistency while eliminating the speed penalty associated with lock-based systems
Solution Approach 2:
The patent introduces version vectors as an intermediary mechanism to track data modifications across nodes. Rather than directly coordinating access through locks, the system uses version vectors to detect conflicts and resolve them asynchronously, enabling faster operations while maintaining consistency through the intermediary conflict resolution layer
2Reliability
If lock-based systems are implemented to coordinate access to shared data among multiple nodes, then data consistency is maintained, but system complexity increases
Solution Approach 1:
The patent removes the complex lock management infrastructure from the distributed database system. By replacing locks with CRDTs and version vectors, the system eliminates the need for centralized coordination, lock acquisition/release protocols, and deadlock detection mechanisms, thereby reducing system complexity while maintaining data consistency
Solution Approach 2:
The patent implements self-service conflict resolution through CRDTs that automatically detect and resolve conflicts without external coordination. Each node independently manages its own data versions using version vectors and applies conflict resolution rules locally, eliminating the need for complex centralized lock management and reducing overall system complexity
3Productivity
If read replica models are used to scale read processing, then read capacity is increased, but data propagation lag occurs due to asynchronous log shipping
Solution Approach 1:
The patent uses CRDTs as a copying mechanism where each node maintains a local copy of the data with version information. Rather than asynchronously shipping SQL logs to replicas, the system copies data with embedded version vectors that enable each node to independently determine its synchronization state and apply updates without lag, maintaining read capacity while eliminating propagation delays
Solution Approach 2:
The patent applies preliminary action by pre-tagging data with version vectors before replication. This allows receiving nodes to immediately process and apply updates in the correct order based on the pre-computed version information, eliminating the need for asynchronous log shipping and reducing data propagation lag while maintaining high read capacity
Data Source
AI summary
A client host may be used to provide access to a shared storage. The client host may receive a read request from a local client for particular data of the shared storage. In response to the read request, the client host may obtain the particular data from a local storage device. The client host may receive a write request from the local client for the shared storage. In response to the write request, the client host may send data to a network-based stream service as one or more stream events for the shared storage. After sending the one or more stream events to the network-based stream service, the client host may receive, from the network-based stream service, an ordered stream event for the shared storage. Based at least in part on the ordered stream event, the client host may update the data stored at the local storage device.