Event-Driven Data Synchronization Between Microservices and Monoliths
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transitioning from monolithic systems to microservice-based architectures is challenging due to issues with data consistency and integrity across distributed services, leading to complexity and overhead costs.
Innovation Solution
An event-driven mechanism is employed to synchronize data across monolithic applications and microservices, using a system of record (SOR) and system of reference (SoREF) to propagate updates, with a distributed cache and outbox table to manage transactions and ensure data integrity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is distributed across microservices to improve scalability and flexibility, then system adaptability improves, but data consistency and integrity deteriorate
Solution Approach 1:
An event bus is introduced as an intermediary component between microservices and the monolithic application. Events are published to the event bus, which then distributes them to subscribed components including the monolithic application and other microservices. This mediator ensures that all components receive and process the same events in a consistent manner, maintaining data integrity while allowing distributed architecture.
Solution Approach 2:
The system uses an outbox table mechanism where events are inserted into the outbox table as a preliminary action before being published to the event bus. This ensures that events are recorded in a transactionally consistent state before distribution, preventing data loss and maintaining consistency across the distributed system.
2Reliability
If a monolithic architecture is used to maintain data consistency, then data integrity improves, but system flexibility and scalability deteriorate
Solution Approach 1:
The system is segmented into independent microservices that can be developed, deployed, and scaled separately from the monolithic application. Each microservice manages its own data independently while publishing events to the event bus for synchronization. This segmentation maintains data integrity through event-driven contracts while enabling flexible independent evolution of system components.
3Reliability
If synchronous data propagation is used to ensure immediate consistency, then data consistency improves, but system performance and resilience deteriorate
Solution Approach 1:
The system uses asynchronous event publishing where microservices publish events to the event bus without waiting for confirmation of propagation to all subscribers. The event bus processes and distributes events periodically in the background. This periodic action maintains data consistency while avoiding synchronous blocking, improving system performance and resilience.
4Productivity
If distributed caching is implemented to improve read performance, then system productivity improves, but data consistency complexity increases
Solution Approach 1:
The monolithic application maintains a local copy (cache) of frequently accessed data from microservices. This copy is updated asynchronously through event subscriptions to the event bus. The copying mechanism improves read performance by serving requests from local cache while event-driven updates maintain consistency, reducing the need for complex real-time synchronization protocols.
Data Source
AI summary
Systems and methods for synchronizing distributed microservices and monolithic systems.