Event-Based Source Replication Architecture for Management Services
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Information technology administrators face challenges in managing the evolving landscape of management service data sources, leading to manual incompatibility corrections and significant financial and time losses due to the need for frequent updates and integrations with external services.
Innovation Solution
An extensible event-based source replication architecture is provided, utilizing a public schema, messaging platforms like Apache Kafka, and an orchestrator to abstract internal management data source microservices, enabling consistent and efficient access to data across multiple data sources and external services.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual incompatibility corrections are developed for every data source modification, then compatibility is maintained, but development time and financial losses increase significantly
Solution Approach 1:
The patent uses event schemas as templates that define the structure and format of events. Instead of manually correcting incompatibilities for each data source, the system copies the standardized event schema format across all data sources, allowing automated parsing and processing without manual intervention.
Solution Approach 2:
The event schema serves as a universal interface that works across multiple data sources and external services. By defining a common event structure that all data sources must follow, the system eliminates the need for source-specific compatibility corrections, as the schema acts as a universal contract for event exchange.
2Reliability
If extensive testing is performed before deployment, then incompatibilities are detected earlier, but testing time and resource costs increase
Solution Approach 1:
The event schema is defined and validated in advance before actual event processing begins. The schema validation rules are established beforehand, so when events are generated from data sources, they are automatically checked against the pre-defined schema, eliminating the need for extensive post-deployment testing.
Solution Approach 2:
The system implements automated schema validation that provides immediate feedback when events do not conform to the expected format. This real-time validation mechanism detects incompatibilities as events are generated, rather than requiring batch testing after deployment, reducing both testing time and resource costs.
3Adaptability or versatility
If the architecture is made extensible to accommodate new data sources, then adaptability improves, but system complexity increases
Solution Approach 1:
The system segments the architecture into distinct components: event sources, event schemas, event validators, and event consumers. Each component has a specific responsibility, and new data sources can be added by simply implementing the event generation interface without modifying the core system logic, thus maintaining simplicity while improving extensibility.
Solution Approach 2:
The event schema acts as an intermediary layer between diverse data sources and the processing system. Instead of creating complex integration logic for each new data source, the schema serves as a standardized mediator that translates various source formats into a common structure, simplifying the architecture while enabling easy extension to new sources.
Data Source
AI summary
Examples of an extensible event based source replication architecture for management services are described. In some examples, an orchestrator service identifies management data source services and a particular management data source service maintains a corresponding database. Network-service-specific replication pipelines are generated for a set of network services. A particular replication pipeline is registered in association with one or more tenant identifiers and one or more entity identifiers. Event data is received as a number of database update messages are received. A subset of the event data is identified according to the entity identifier and the tenant identifier and is replicated into the particular replication pipeline for a particular network service.