Multi-Tenant Transactional Outbox for Ordered Event Publishing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing transactional outbox patterns for multi-tenant microservices face challenges in reliable, resilient, and ordered event publishing, especially when interacting with multiple databases or schemas, leading to complex setups and excessive overhead due to polling and change data capture requirements.
Innovation Solution
A notification-based system is implemented where a core execution thread publishes a notification to an event publisher upon a successful atomic transaction, allowing the publisher to query and publish events from an event outbox in the correct order, using a real-time streaming platform to ensure efficient and isolated event delivery across tenants.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If polling-based transactional outbox pattern is used in multi-tenant microservices, then event publishing can be achieved, but system complexity and overhead increase significantly
Solution Approach 1:
The patent introduces an event relay table as an intermediary component between the transactional outbox and the event publishing mechanism. This relay table acts as a mediator that receives notifications from the core execution thread and forwards them to the event publisher, simplifying the overall system architecture by eliminating the need for complex polling mechanisms while maintaining event publishing reliability
Solution Approach 2:
The patent implements a notification-based feedback mechanism where the core execution thread publishes notifications to the event relay table upon successful atomic transactions. This feedback loop enables the event publisher to be notified and publish events from the outbox in the correct order without requiring frequent polling, thus reducing system complexity while maintaining reliability
2Reliability
If frequent polling is implemented to ensure event publishing, then event delivery reliability improves, but latency and overhead increase
Solution Approach 1:
The patent replaces frequent periodic polling with event-driven periodic action through notifications. The core execution thread publishes notifications to the event relay table only when atomic transactions succeed, triggering the event publisher to act at appropriate moments rather than through continuous or frequent polling. This reduces polling latency while maintaining event delivery reliability
Solution Approach 2:
The notification mechanism provides immediate feedback to the event publisher when events are available for publishing. This feedback-based approach eliminates the need for frequent polling by enabling the publisher to respond only when necessary, thus reducing time loss while ensuring reliable event delivery
3Reliability
If change data capture is used for event publishing, then event consistency is maintained, but setup complexity and overhead increase
Solution Approach 1:
The event relay table serves as an intermediary that simplifies the event publishing process by receiving notifications directly from the core execution thread. This eliminates the need for complex change data capture (CDC) setup while maintaining event consistency through the notification-based mechanism that ensures events are published in the correct order
Solution Approach 2:
The patent extracts the complexity of CDC setup by introducing a simpler notification-based mechanism. Instead of implementing full CDC infrastructure, the system extracts only the essential functionality through the event relay table and notification mechanism, reducing setup complexity while maintaining event consistency
4Reliability
If SAGA orchestrator service is added for event coordination, then event ordering is improved, but system complexity increases
Solution Approach 1:
The event relay table performs multiple functions: it receives notifications from the core execution thread, maintains event ordering information, and enables the event publisher to publish events in the correct order. This multi-functional component eliminates the need for a separate SAGA orchestrator service, reducing system complexity while maintaining event ordering reliability
Solution Approach 2:
The patent merges the orchestration functionality into the existing event publishing infrastructure through the event relay table. Instead of adding a separate SAGA orchestrator service, the relay table combines notification management, event ordering, and publishing coordination into a single component, thereby reducing orchestration complexity while ensuring correct event ordering
5Reliability
If tenant data isolation is maintained across multiple databases, then data security is improved, but event publishing complexity increases
Solution Approach 1:
The patent segments the event publishing process by introducing the event relay table that handles notifications from multiple tenant databases independently. Each tenant's events are processed through the same relay table mechanism, maintaining data isolation while simplifying the multi-database setup through a unified event publishing approach
Solution Approach 2:
The event relay table and notification mechanism serve as a universal solution that works across multiple tenant databases and schemas. This multi-functional approach maintains tenant data isolation while eliminating the need for separate event publishing setups for each database, thereby reducing overall system complexity
Data Source
AI summary
Operations include initiating execution of an atomic transaction; receiving confirmation of execution of the atomic transaction; publishing a notification corresponding to the execution of the atomic transaction; receiving the notification corresponding to the execution of the atomic transaction; retrieving the information corresponding to the database transaction; and publishing an event based on the information corresponding to the database transaction. The atomic transaction comprises (a) executing a database transaction adding, modifying, or deleting a value in a database instance and (b) logging, to an event outbox associated with a particular tenant among multiple tenants, information corresponding to the database transaction. The notification comprises information that can be used to identify the event outbox storing the information corresponding to the database transaction.