Flexible Data Pipeline With Dynamic Schema Routing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing systems fail to effectively address the need for processing data from various sources, particularly in the context of addressing data schema mismatches, and the need for a flexible data pipeline that transforms the data pipeline that transforms the data received from the data sources into a format expected by consumers, without the need for a flexible data pipeline that is able to process any data to be processed by the data pipeline that is expected by the data pipeline that is expected by the data pipeline that is expected by the data pipeline that is expected by the data pipeline that is expected by the data pipeline.
Innovation Solution
The apparatus comprises a router configured to receive data from the data pipeline that transforms the data pipeline that is expected by the data pipeline.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Stability of the object's composition
If a fixed data schema (e.g., ISA-95) is used for data storage, then data consistency and standardization are improved, but the system cannot accommodate custom or evolving data formats without stopping the data pipeline
Solution Approach 1:
The data pipeline is segmented into multiple independent domain modules (generic domain module and custom domain module), each handling specific data schemas. This segmentation allows the system to maintain a stable generic schema while accommodating custom schemas through separate modules, resolving the contradiction between schema consistency and format flexibility.
Solution Approach 2:
The system dynamically routes data messages to appropriate domain modules based on the message type and schema requirements. The router dynamically determines whether to use the generic domain module or custom domain module, enabling the system to adapt to different data formats without changing the overall pipeline structure or stopping operations.
2Adaptability or versatility
If the data pipeline is stopped to deploy new schemas or databases, then data storage adaptability is improved, but data loss occurs and productivity decreases
Solution Approach 1:
The custom domain module is deployed in advance with predefined custom schemas and transformation rules. This preliminary action allows the system to immediately handle custom data formats when they arrive, without needing to stop the pipeline for schema deployment, thus maintaining both adaptability and productivity.
Solution Approach 2:
The data pipeline maintains continuous operation by routing different message types to different domain modules simultaneously. The router ensures that custom messages are directed to the custom domain module while generic messages go to the generic domain module, allowing the system to process diverse data formats without interruption and prevent data loss.
3Adaptability or versatility
If multiple databases with different schemas are maintained, then data format versatility is improved, but system complexity increases
Solution Approach 1:
The router acts as an intermediary that manages the complexity of multiple databases and schemas. It receives incoming messages, determines the appropriate target database and domain module based on message type, and routes messages accordingly. This intermediary simplifies the system architecture by centralizing the decision-making logic and shielding the complexity from the rest of the system.
Solution Approach 2:
The router serves multiple functions: message reception, schema identification, database selection, and domain module routing. This multi-functionality reduces the need for separate components for each task, thereby managing system complexity while supporting multiple data formats and databases.
4Adaptability or versatility
If custom domain modules are deployed dynamically, then data pipeline flexibility is improved, but the risk of message routing errors increases
Solution Approach 1:
The system uses feedback mechanisms where the router receives information about message types and schemas, determines the appropriate custom domain module, and routes messages accordingly. The cache module stores mappings between schemas and databases, providing feedback information that ensures accurate routing. This feedback loop maintains reliability while enabling dynamic deployment of custom domain modules.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
An apparatus acting as an entry point to a cloud provider network (CPN) connected to a generic database (GDB) able to store generic messages and to a custom database (CDB) able to store custom messages, the apparatus comprising: a router (RT) configured to receive generic messages from a generic data source (GDS) and custom messages from a custom data source (CDS), each of the generic message and the custom message containing a message-type and a payload, a cache module (MD) storing at least an identifier of the generic schema associated with an identifier of the generic database and an identifier of the custom schema associated with an identifier of the custom database, a custom domain module (CDM) able to deserialize the payload of a custom message according to the custom schema and to transfer the custom message in an expected format to the custom database using custom transformation rules, wherein the custom transformation rules are defined after the deployment of the generic domain module and the generic database, wherein, upon reception of a custom message, the router (RT) is configured to read the message-type to extract an identifier of a custom schema, to interrogate the cache module with the identifier of the custom schema to determine the identifier of the custom database (CDB) associated with the extracted identifier of the custom schema and to transfer the custom message to the custom domain module, wherein the cache module (MD) is updated with an identifier of the custom schema associated with an identifier of the custom database (CDB) after the deployment of the custom domain module (CDM).