Flexible Data Pipeline Routing for Hot Schema Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Industrial data pipelines face challenges in processing and storing data from various sources without interrupting the data pipeline execution, especially when data does not conform to the ISA-95 standard, leading to issues with serialization and storage, resulting in data loss during schema updates and deployments.
Innovation Solution
A system comprising a router, cache module, generic, and custom domain modules that deserialize and transform messages according to predefined and custom schemas, allowing for seamless storage in generic and custom databases without interrupting the data pipeline, using predefined generic transformation rules and custom transformation rules defined after deployment, and utilizing a buffer layer for live or batch data processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data pipeline uses standard ISA-95 schema for storing all data, then database compatibility and standardization are improved, but data loss occurs when custom data types (e.g., OEE values) cannot be serialized with the standard schema
Solution Approach 1:
The patent divides the database into two separate schemas: a standard ISA-95 schema for conventional data and a custom schema for specialized data types like OEE values. This segmentation allows each schema to be optimized for its specific purpose, preventing data loss while maintaining standardization where applicable.
Solution Approach 2:
The patent introduces a message router as an intermediary component that receives messages, identifies their type, and routes them to the appropriate database schema. This mediator enables seamless handling of both standard and custom data types without requiring the entire system to accommodate every data variant.
2Adaptability or versatility
If data pipeline is stopped for schema updates and deployments, then database evolution and new data type support are improved, but data loss occurs during stop/start cycles and productivity decreases
Solution Approach 1:
The patent pre-configures multiple database schemas (standard and custom) before they are needed. When custom data types need to be supported, the corresponding schema and message router rules are already in place, allowing immediate routing without stopping the data pipeline for schema creation.
Solution Approach 2:
The patent implements a dynamic message routing system where the router can identify and route messages to different database schemas based on message type. This dynamic capability allows the system to adapt to new data types in real-time without interruption to the data pipeline execution.
3Adaptability or versatility
If custom domain module is deployed after generic database deployment, then support for custom data types is improved, but system complexity increases with multiple schemas and routing logic
Solution Approach 1:
The message router is designed as a universal component that handles both standard ISA-95 messages and custom data type messages through a single interface. It uses message type identification to dynamically route to the appropriate schema, providing multi-functionality without requiring separate routing systems for each data type.
Solution Approach 2:
The patent creates a custom schema that mirrors the structure of the standard ISA-95 schema but is tailored for custom data types like OEE values. This copying approach allows the custom schema to follow similar organizational patterns and access methods as the standard schema, reducing the operational complexity despite the additional schema.
Data Source
AI summary
An apparatus acting as an entry point to a cloud provider network connected to a generic database able to store generic messages and to a custom database able to store custom messages. The apparatus includes: a router configured to receive generic messages from a generic data source and custom messages from a custom data source, each of the generic message and the custom message containing a message-type and a payload; a cache module storing at least an identifier of the generic schema associated with an identifier of the generic database and an identifier of the custom schema associated with an identifier of the custom database; and a custom domain module able to deserialize the payload of a custom message according to the custom schema and to transfer the custom message in an expected format to the custom database using custom transformation rules, wherein the custom transformation rules are defined after the deployment of the generic domain module and the generic database. Upon reception of a custom message, the router is configured to read the message-type to extract an identifier of a custom schema, to interrogate the cache module with the identifier of the custom schema to determine the identifier of the custom database associated with the extracted identifier of the custom schema and to transfer the custom message to the custom domain module. The cache module is updated with an identifier of the custom schema associated with an identifier of the custom database after the deployment of the custom domain module.