Visual Data Pipeline Mapping for External ML Model Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Configuring industrial data pipelines is a complex task that requires expertise, and integrating external analytic models into enterprise-specific data pipelines can be challenging due to lack of knowledge about the end user's data or schema and naming conventions.
Innovation Solution
A data pipeline configuration system that allows intuitive visual configuration of pipelines using a graphical interface, enabling users to select and link pipeline components, map data to analytic models, and publish results to specified destinations, with support for importing and configuring machine learning models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If external analytic models are integrated into enterprise data pipelines, then the analytical capability and versatility of the pipeline is improved, but the complexity of configuration and integration increases due to lack of knowledge about data sources and schema
Solution Approach 1:
The patent introduces an intermediary component that automatically discovers and maps data sources to analytic model inputs. This intermediary layer handles the complexity of integration by providing automated schema matching and data flow configuration, allowing external models to be integrated without requiring deep knowledge of the enterprise data infrastructure.
Solution Approach 2:
The system enables self-service integration through automated model discovery and configuration. The pipeline automatically detects available data sources, matches them to model inputs based on schema compatibility, and configures the integration without manual intervention, reducing the complexity burden on users.
2Ease of operation
If automated model discovery and mapping is implemented, then the ease of operation is improved, but the system complexity increases due to additional discovery and mapping components
Solution Approach 1:
The patent merges the model discovery, data source detection, and mapping functions into an integrated automated configuration system. By combining these functions into a unified process, the system reduces operational complexity while managing the underlying system architecture complexity through consolidation rather than separate components.
Solution Approach 2:
The system performs preliminary actions by automatically discovering data sources and mapping them to model inputs before the user needs to configure the pipeline. This preliminary automated configuration reduces the operational effort required while the system complexity is managed through pre-computed mappings and cached discovery results.
Data Source
AI summary
A data pipeline configuration system allows industrial data pipelines to be configured using an intuitive visual interface. The pipeline configuration system allows graphical pipeline components representing data sources, data processing, analytic or machine learning models, and emitters to be selectively added to an industrial data pipeline application by selecting these components from a library. The pipeline configuration application is created by arranging and linking these selected pipeline components within a pipeline builder section of the configuration system's visual design interface. The design interface also allows analytic or machine learning models to be easily integrated into the pipeline application and mapped to incoming data items, such that the model is applied and scored against incoming data during pipeline operation. The configuration system also allows the user to configure destinations or data sinks for the pipeline data, including both the incoming industrial data and model scoring results.