Provisional Authority in Distributed Database Nodes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Centralized distributed databases face scalability issues due to high bandwidth requirements and latency, while partitioned databases suffer from inconsistencies and lack of reliable data consistency between nodes, especially in highly distributed systems.
Innovation Solution
Implementing a distributed database system where members can provisionally write data locally without consulting the master, with subsequent reconciliation, using semantic replication and atomic transactions to ensure data consistency and reduce latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all writes go through the master in a centralized distributed database, then data consistency is maintained, but latency increases and the master becomes a bottleneck
Solution Approach 1:
The system performs preliminary actions by having members apply writes locally to their version of the database before the write is confirmed by the master. This preliminary local application reduces latency while the master subsequently reconciles the write to ensure consistency across the distributed system.
Solution Approach 2:
The write operation is segmented into two phases: local application at the member and subsequent reconciliation at the master. This segmentation allows the time-critical local operation to proceed independently while the consistency-critical reconciliation happens afterward, resolving the contradiction between speed and consistency.
2Reliability
If all changed data is transmitted to members in a centralized database, then data is replicated across nodes, but bandwidth is consumed and the system can be overwhelmed
Solution Approach 1:
Each member maintains its own local version of the database and applies writes locally without receiving full data transmissions from the master. This local quality approach eliminates the need for continuous data transmission while maintaining data availability, dramatically reducing bandwidth consumption.
Solution Approach 2:
The system extracts the data transmission step from the write operation. Instead of transmitting data to members for every write, the member independently applies the write locally and only transmits the write request itself to the master for reconciliation, eliminating unnecessary data movement.
3Reliability
If a single overall master is selected to coordinate writes in a partitioned database, then data consistency between partitions is maintained, but the master becomes a bottleneck
Solution Approach 1:
The master acts as an intermediary that receives write requests from members and provides conflict resolution information back to members. This intermediary role allows distributed members to perform writes independently while the master coordinates only when necessary, maintaining consistency without becoming a throughput bottleneck.
Solution Approach 2:
The system implements feedback by having the master send conflict resolution information back to members after receiving their write requests. This feedback mechanism enables members to adjust their writes based on master coordination, maintaining consistency while preserving high write throughput through independent local operations.
4Productivity
If data is not reliably consistent between active and passive nodes in an HA pair, then system availability is maintained, but data loss can occur upon node failure
Solution Approach 1:
The system merges the active and passive nodes into a single logical write target. Writes are applied to both nodes as a single atomic transaction, ensuring that both nodes have identical data. This merging maintains system availability while eliminating data loss risk through guaranteed consistency between HA nodes.
Data Source
AI summary
Techniques for writing data in a distributed database system having a plurality of members and a master are disclosed. In one embodiment, a write request is received at an originating member. The write request is applied to a version of a database on the originating member and is placed in a journal queue of items for propagating to the master in one single atomic action. The write request is propagated to the master. Whether the propagated write request is associated with a conflict is determined, and in an event it is determined that the propagated write request is associated with the conflict, an indication that the write request is associated with the conflict is sent to the originating member.