ETL Workflow Recommendation via Knowledge Graph Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current ETL workflow systems lack the ability to recommend specific workflows based on user inputs related to predetermined business domains, leading to inefficiencies in data preparation, cleaning, and analysis, and do not effectively consider policy and access conditions.
Innovation Solution
An ETL workflow recommendation device that includes a knowledge database storing an overall knowledge graph, an input management unit for converting user inputs into graph queries, an ETL recommendation unit for generating workflow candidates, and an output management unit for evaluating and reporting recommended workflows, ensuring compliance with policy and access conditions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If ETL workflows are manually programmed according to each data source, then data aggregation can be achieved, but development man-hours increase significantly
Solution Approach 1:
The system enables self-service by allowing users to input natural language queries and automatically generating ETL workflows without manual programming. The AI model processes user requests, searches knowledge graphs, and generates appropriate ETL job definitions automatically, eliminating the need for developers to manually program each ETL workflow according to data source requirements.
Solution Approach 2:
The patent replaces the mechanical system of manual ETL workflow programming with an AI-based automated system. Instead of requiring developers to write and configure ETL jobs manually, the system uses natural language processing and AI models to automatically generate ETL workflows from user queries, substituting human expertise with automated intelligent systems.
2Reliability
If specialized knowledge is required for each data source, then data can be processed accurately, but the complexity of programming increases
Solution Approach 1:
The patent introduces knowledge graphs as intermediaries between users and data sources. The knowledge graph stores structured information about data sources, their schemas, and relationships. When users make queries, the system searches the knowledge graph to automatically determine the appropriate ETL workflows, eliminating the need for users to have specialized knowledge about each data source while maintaining processing accuracy.
Solution Approach 2:
The system incorporates feedback mechanisms where the AI model continuously learns from user interactions and updates its understanding of data sources. This feedback loop enables the system to improve its automated workflow generation over time, maintaining high accuracy while reducing the complexity burden on users.
3Ease of manufacture
If ETL workflows are generated without considering policy conditions, then workflow generation is simple, but compliance with business requirements is compromised
Solution Approach 1:
The patent applies preliminary action by pre-configuring policy conditions and access control rules within the system before workflow generation occurs. The knowledge graph and AI model are pre-loaded with business domain-specific policies and constraints, enabling the system to automatically generate compliant workflows without requiring complex real-time policy evaluation logic.
Solution Approach 2:
The system achieves universality by creating a single ETL workflow generation platform that handles multiple business domains and policy requirements simultaneously. The AI model and knowledge graph can adapt to different business contexts and policy frameworks, making the system universally applicable across various organizations and domains while maintaining compliance.
Data Source
AI summary
Aspects relate to recommending ETL workflows for performing specific tasks based on user inputs related to a predetermined business domain while complying with policy and access conditions. Provided is an ETL workflow recommendation device including a knowledge database for storing an overall knowledge graph including an ETL knowledge graph that at least indicates ETL information about a predetermined business domain in a graph format, an input management unit that receives a user input related to the predetermined business domain and converts the user input into a graph query for searching the overall knowledge graph, an ETL recommendation unit that searches the overall knowledge graph using the graph query and generates ETL workflow candidates with respect to the user input; and an output management unit that evaluates the ETL workflow candidates and outputs an ETL workflow report indicating a recommended ETL workflow.