Concurrent Outbox Table for Reliable Microservice Messaging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In modern cloud architecture, microservices face challenges in reliably reporting database actions to external systems due to issues like failed database transactions and application crashes, especially in multitenant databases with shared resources and load balancing.
Innovation Solution
The introduction of a concurrent outbox table integrated into each microservice application, where database actions are written and later processed by an outbox table processor to generate notifications to external systems, ensuring reliable messaging even in the presence of failures or application crashes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If database actions are reported directly to external systems without buffering, then response speed is improved, but reliability deteriorates due to potential application crashes or network failures
Solution Approach 1:
The patent introduces an intermediary outbox table between the microservice application and external systems. The outbox table buffers database action notifications before they are sent to external systems, ensuring that even if the application crashes or network fails, the notifications are not lost. This intermediary layer decouples the direct dependency between the application and external systems, resolving the contradiction between reliability and complexity.
Solution Approach 2:
The outbox table performs preliminary action by pre-storing notifications that need to be sent to external systems. When a database action occurs, the notification is immediately written to the outbox table, ensuring it is prepared and safe even if subsequent sending operations fail. This preliminary buffering action ensures reliability without requiring complex real-time retry mechanisms.
2Productivity
If notifications are sent synchronously to external systems, then response time is improved, but system stability deteriorates due to potential crashes during transmission
Solution Approach 1:
The outbox table performs preliminary action by pre-storing notifications that need to be sent to external systems. When a database action occurs, the notification is immediately written to the outbox table, ensuring it is prepared and safe even if subsequent sending operations fail. This preliminary buffering action ensures reliability without requiring complex real-time retry mechanisms.
Solution Approach 2:
The outbox table acts as an intermediary that decouples the synchronous database transaction from the asynchronous notification sending. The database action triggers a write to the outbox table within the same transaction, while the actual sending to external systems can happen asynchronously, preventing transmission failures from affecting application stability.
3Manufacturing precision
If the outbox table is processed synchronously with database transactions, then data consistency is improved, but system performance deteriorates due to blocking
Solution Approach 1:
The outbox table performs preliminary action by pre-storing notifications that need to be sent to external systems. When a database action occurs, the notification is immediately written to the outbox table, ensuring it is prepared and safe even if subsequent sending operations fail. This preliminary buffering action ensures reliability without requiring complex real-time retry mechanisms.
Solution Approach 2:
The system dynamically separates the database transaction from the notification sending process. The outbox table is written within the database transaction to ensure consistency, while the actual sending to external systems can happen asynchronously when the transaction is complete, preventing blocking and improving throughput.
4Quantity of substance
If multiple microservice instances share common database resources, then resource efficiency is improved, but reliability deteriorates due to race conditions and conflicts
Solution Approach 1:
The patent segments the notification processing by creating separate outbox tables for each microservice instance. Each instance has its own outbox table, which isolates the notifications to be sent by that specific instance. This segmentation prevents race conditions between multiple instances while they share common database resources, as each instance's notifications are independently managed in its own outbox table.
Solution Approach 2:
Each microserviceinstance has its own outbox table with local properties specific to that instance. The outbox table includes instance-specific identifiers and processing logic, ensuring that notifications are correctly attributed and processed by the appropriate instance. This local quality approach maintains reliability in shared database environments by isolating instance-specific state.
Data Source
AI summary
In an example embodiment, a solution is provided in which an outbox table is added to each container in a database. Rather than perform an emit solely using the application instance alone (in response to a notification from the database that the underlying database action has been performed), or using an outside outbox table processor, an outbox table processor is integrated into the microservice application. When a database action is performed by the microservice application, that database action is written into the outbox table in the corresponding container in the database. Furthermore, whenever the outbox table processor determines that the state of the database has changed in a way that leads to an emit, it then reads the actions in the outbox table, issues an emit to notify one or more external systems of the actions, and deletes those actions from the outbox table.