Consensus Key Locking with Local Cache for Idempotent Transactions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing transaction exchange platforms face challenges in guaranteeing idempotent transactions across geographically disparate regions due to issues with duplicate data processing and high latency caused by consensus protocols, which are inefficient and unable to scale effectively.
Innovation Solution
A transaction exchange platform utilizing a streaming data platform and microservices that leverages a consensus protocol only for lock acquisition, employing a local fast write forward system for data access, thereby improving performance and reliability by limiting consensus protocol interactions to unique key writes and storing metadata in a local cache system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If consensus protocol is used for all read and write transactions, then idempotent transaction guarantee is achieved, but latency increases and scalability deteriorates
Solution Approach 1:
The patent segments transaction operations into two categories: lock acquisition operations that require consensus protocol for idempotent guarantee, and data access operations that use local fast write forward system without consensus overhead. This segmentation allows the system to maintain reliability for critical operations while improving performance for routine operations.
Solution Approach 2:
The patent implements local quality by allowing different regions to have different data copies that can be accessed locally without consensus protocol. Each region maintains local state for fast access, while consensus protocol is only invoked when lock acquisition is needed, creating local optimization for performance while preserving global consistency for critical operations.
2Reliability
If consensus protocol is used for all transactions, then idempotent transaction guarantee is achieved, but system scalability worsens
Solution Approach 1:
The patent segments the transaction processing workload by separating lock acquisition (requiring consensus) from data access (using local system). This allows the system to scale horizontally by adding more regions with local data copies, as not all operations require consensus coordination, thereby improving scalability while maintaining idempotent guarantees for critical operations.
Solution Approach 2:
The patent applies partial action by using consensus protocol only when necessary (for lock acquisition) rather than for all transactions. This selective application of consensus reduces the coordination overhead and allows the system to scale more effectively, while still providing idempotent guarantees where needed.
3Loss of time
If local fast write forward system is used for data access, then latency is reduced, but risk of duplicate processing increases
Solution Approach 1:
The patent applies preliminary action by acquiring locks before performing data access operations. The lock acquisition phase uses consensus protocol to establish exclusive rights to modify data, preventing duplicate processing. Once the lock is acquired, the local fast write forward system can safely access and modify data without risk of duplicates, as other regions are blocked by the consensus-based lock.
Solution Approach 2:
The patent introduces locks as an intermediary mechanism between the local fast write forward system and the consensus protocol. The lock acts as a mediator that allows the local system to operate independently and quickly while still being protected from duplicate processing by the consensus-based locking mechanism. This intermediary enables the coexistence of fast local access and duplicate prevention.
Data Source
AI summary
Aspects described herein may relate to a transaction exchange platform using a streaming data platform (SDP) and microservices to process transactions according to review and approval workflows. The transaction exchange platform may receive transactions from origination sources, which may be added to the SDP as transaction objects. As the transactions are processed, the transactions may require access to a resource (e.g., a key value in a database). A microservice processing the transaction may request, from a locking microservice, a lock for the resource. The locking microservice may query a local cache to determine whether a lock exists for the resource. If the local cache determines that no lock exists for resource, the locking mechanism may employ a consensus protocol to obtain a lock for the resource from a plurality of clusters. If consensus is reached, a lock for the resource may be granted to the requesting microservice.