Consensus Key Locking With Local Cache for Idempotent Transactions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing transaction exchange platforms struggle to guarantee idempotent transactions across geographically disparate regions due to issues with traditional write forward mechanisms, consensus protocols, and database solutions that incur high latency and overhead, leading to potential duplicate processing and limited data synchronization.
Innovation Solution
Implement a transaction exchange platform using a streaming data platform and microservices, where a consensus protocol is leveraged only for lock acquisition, and a fast write forward system handles data access, ensuring idempotent transactions by storing metadata and application state details in a local cache after lock acquisition.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a consensus protocol is used to guarantee idempotent transactions across all regions, then transaction reliability is improved, but latency increases due to consensus overhead
Solution Approach 1:
The patent segments the locking mechanism into two parts: a distributed consensus component for acquiring locks on key values across regions, and a local write-forward component for subsequent data access. This segmentation allows the system to achieve consensus only when needed (during lock acquisition) while performing fast local operations for the majority of transactions, thereby reducing overall latency while maintaining reliability.
Solution Approach 2:
The patent implements preliminary lock acquisition through consensus protocol before executing the actual transaction processing. By acquiring the lock in advance and storing metadata in a local cache system, the system ensures that subsequent operations can proceed locally without repeated consensus calls, reducing latency for the bulk of transactions while maintaining the reliability guarantees of consensus-based locking.
2Stability of the object's composition
If a consensus protocol is used for all read and write transactions, then data consistency is improved, but system performance deteriorates due to excessive overhead
Solution Approach 1:
The patent applies partial action by using the consensus protocol only for the specific operation of acquiring locks on key values, rather than for all read and write transactions. Once the lock is acquired through consensus, the system performs subsequent data access operations locally without requiring additional consensus calls. This partial application of consensus maintains data consistency where needed while significantly improving overall system performance.
Solution Approach 2:
The patent introduces a local cache system as an intermediary between the distributed consensus locking mechanism and the actual data storage. The cache stores metadata about locked key values, allowing the system to perform fast local read and write operations without repeatedly querying the distributed consensus system. This intermediary enables the system to achieve both consistency (through consensus-based locking) and performance (through local cache operations).
3Productivity
If traditional write forward mechanisms are used, then processing speed is improved, but duplicate data processing occurs due to window of inconsistency
Solution Approach 1:
The patent performs the lock acquisition action in advance, before the write-forward processing begins. By acquiring the lock through consensus protocol and storing the locked key value metadata in a local cache system before executing the transaction, the system ensures that no other instance can process the same key value simultaneously. This preliminary locking action eliminates the window of inconsistency that leads to duplicate processing, while still allowing fast local write-forward operations to proceed.
4Reliability
If relational databases are used for frequent conflict resolution, then transaction handling is improved, but performance issues occur due to replication overhead
Solution Approach 1:
The patent segments the transaction handling into two distinct phases: lock acquisition through distributed consensus (handling conflict resolution) and local write-forward processing (handling data operations). By separating these functions, the system can use efficient consensus algorithms for conflict resolution without subjecting all operations to the same overhead. The local cache system further segments the data access pattern, allowing frequent local operations to proceed without replication overhead.
Solution Approach 2:
The patent introduces a local cache system as an intermediary that stores metadata about locked key values. This cache acts as a mediator between the distributed consensus locking mechanism and the local data processing operations. By caching the results of consensus-based lock acquisition, the system can perform fast local operations without repeatedly engaging the replication and conflict resolution mechanisms of relational databases, thereby improving performance while maintaining reliable transaction handling.
Data Source
AI summary
Aspects described herein may relate to a transaction exchange platform using a streaming data platform (SDP) and microservices to process transactions according to review and approval workflows. The transaction exchange platform may receive transactions from origination sources, which may be added to the SDP as transaction objects. As the transactions are processed, the transactions may require access to a resource (e.g., a key value in a database). A microservice processing the transaction may request, from a locking microservice, a lock for the resource. The locking microservice may query a local cache to determine whether a lock exists for the resource. If the local cache determines that no lock exists for resource, the locking mechanism may employ a consensus protocol to obtain a lock for the resource from a plurality of clusters. If consensus is reached, a lock for the resource may be granted to the requesting microservice.