Automated ETL Workflow Generation via Knowledge Graph Ontology
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing ETL systems are inaccessible to non-experts, limiting their potential impact and usability, as they require specialized expertise for data extraction, transformation, and loading processes.
Innovation Solution
A system and method utilizing a knowledge graph representing an ETL-based ontology to automate the generation and execution of ETL workflows, allowing users to receive ETL workflows based on data analysis requests, incorporating data lineage, access rules, and optimization techniques for efficient data processing and storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If ETL systems are made accessible to non-experts, then ease of operation is improved, but device complexity increases
Solution Approach 1:
The patent introduces a knowledge graph as an intermediary layer between users and the complex ETL system. The knowledge graph translates user-friendly data requests into complex ETL workflows automatically, shielding users from system complexity while maintaining functionality. This mediator enables non-experts to access ETL capabilities without needing to understand the underlying complexity.
Solution Approach 2:
The system implements self-service through automated workflow generation. When users submit data requests, the system automatically queries the knowledge graph, determines appropriate ETL workflows, and executes them without requiring user expertise or manual configuration. This automation allows non-experts to perform complex data operations independently.
2Productivity
If automated ETL workflow generation is implemented, then productivity is improved, but device complexity increases
Solution Approach 1:
The knowledge graph is pre-built and pre-configured with ontologies, data lineage relationships, and governance rules before user requests arrive. This preliminary preparation enables the system to automatically generate optimized ETL workflows without requiring complex real-time decision-making, thereby improving productivity while managing complexity through pre-computed structures.
Solution Approach 2:
The system incorporates feedback mechanisms where execution results and data quality metrics are fed back into the knowledge graph. This feedback loop continuously refines the ontology and improves automated workflow generation accuracy over time, enhancing productivity while the structured feedback process manages system complexity through iterative optimization.
Data Source
AI summary
The exemplary embodiments disclose a system and method, a computer program product, and a computer system. The exemplary embodiments may include receiving a data analysis request, using a knowledge graph for determining a source dataset based on the received data analysis request, wherein the knowledge graph represents an extract, transform and load (ETL) based ontology, wherein the knowledge graph comprises nodes representing entities and edges representing relationships between the entities, and wherein the entities are instances of concepts of the ETL based ontology, building an ETL workflow for processing the source dataset in accordance with a data lineage associated with the source dataset in the knowledge graph, and executing the ETL workflow.