Automated Data Integration Pipeline for Industrial Ontology Mapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data integration platforms, such as iPaaS and ETL/ELT solutions, are inadequate for industrial and commercial automation systems due to their manual configuration requirements, inability to handle heterogeneous systems, and slow deployment processes, especially in environments with tens of thousands of devices across multiple protocols and locations.
Innovation Solution
A platform that automatically discovers, extracts, maps, merges, and enriches data from on-premises and cloud systems using a data source discovery mechanism, data extraction system, data mapping mechanism, data storage system, and data merging method, providing normalized and enriched data through APIs and real-time streams.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual configuration of data sources and field mapping is used in existing iPaaS and ETL/ELT solutions, then data integration can be achieved, but the deployment time increases to months and the process becomes too complex for enterprise-wide deployment
Solution Approach 1:
The system performs automatic data source discovery through introspection and network scanning, automatically maps fields using machine learning algorithms, and configures integration pipelines without human intervention. This self-service capability eliminates manual configuration steps while maintaining integration accuracy through automated validation and pattern recognition.
Solution Approach 2:
The system pre-configures standardized data models, field mappings, and integration templates that can be automatically applied to new data sources. By preparing these resources in advance, the system eliminates the need for lengthy manual configuration during deployment, reducing deployment time from months to minutes while preserving data integration reliability.
2Adaptability or versatility
If existing iPaaS and ETL/ELT solutions are used for industrial and commercial automation systems with tens of thousands of devices, then data integration is possible, but the manual mapping of fields results in bespoke output formats that are not useful beyond the creator's target
Solution Approach 1:
The system implements a universal standardized data model that can represent multiple data sources and protocols through a single consistent schema. This standardized model serves as a common interface for all data integration operations, eliminating the need for bespoke output formats for each target system and enabling broad reusability across different applications and platforms.
Solution Approach 2:
The system dynamically adjusts data transformation parameters based on the standardized model requirements rather than target-specific parameters. By changing the approach from target-customized transformations to model-standardized transformations, the system simplifies output formats while maintaining adaptability to handle diverse input sources through automatic parameter mapping and protocol translation.
3Manufacturing precision
If manual field mapping is performed in current data integration solutions, then data can be transformed, but the output data format becomes bespoke and not useful beyond the creator's target system
Solution Approach 1:
The system pre-defines standardized data models with explicit field definitions, data types, and validation rules that serve as templates for all mapping operations. By establishing these standards in advance, the system ensures consistent and accurate mapping across all data sources while producing uniform output formats that can be reused across multiple targets, eliminating the need for repeated manual mapping for each application.
Solution Approach 2:
The system creates reusable mapping templates and data transformation patterns that can be copied and applied to multiple data sources and targets. Instead of creating custom mappings for each scenario, the system leverages standardized templates that capture proven mapping logic, ensuring consistent accuracy while enabling broad reusability across different data integration scenarios without requiring manual reconfiguration.
Data Source
AI summary
Described are platforms, systems, and methods to automatically discover, extract, map, merge, and enrich data found in on-premises in automated industrial and commercial environments and cloud systems for purposes of providing developers access to normalized, merged, and enriched data through an API. The platforms, systems, and methods identify a plurality of data sources associated with an automation environment; retrieve data from at least one of the identified data sources; apply a first algorithm to map the retrieved data to a predetermined ontology; merge the mapped data into a data store comprising timeseries of the mapped data; apply a second algorithm to identify patterns in the merged data and enriching the data based on one or more identified patterns; and provide one or more APIs or one or more real-time streams to provide access to the enriched data.