Data Transformation Graphs for Automatic Workflow Dependency Resolution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data transformation pipeline management systems require manual user intervention to define causal relationships and metadata, making it difficult to manage complex workflows and adapt to different environments, and are cumbersome to set up and deploy.
Innovation Solution
A method for generating a data transformation graph automatically based on defined data transformations, allowing modular definition of inputs and outputs without explicit data flow, and enabling execution in various environments with minimal setup.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If manual workflow management platforms (Airflow, Luigi, Foundry) are used to manage data transformations, then workflow orchestration and scheduling can be achieved, but the system complexity increases and requires cumbersome setup with scheduler servers
Solution Approach 1:
The system enables self-service workflow execution by allowing data transformations to be defined as modular functions with automatic dependency resolution. The workflow engine automatically discovers transformations, resolves dependencies between them, and executes them in the correct order without requiring manual orchestration configuration or scheduler servers.
Solution Approach 2:
The patent extracts the workflow orchestration logic from complex centralized scheduler servers and embeds it directly in the data transformation definitions themselves. Each transformation function contains its own metadata about dependencies, inputs, and outputs, allowing the system to self-orchestrate without external control infrastructure.
2Reliability
If explicit causal relationships and metadata are defined manually for each workflow step, then workflow execution can be controlled, but the ease of operation decreases and makes it difficult to manage complex workflows
Solution Approach 1:
Transformations automatically declare their own dependencies, inputs, and outputs through function metadata and type annotations. The workflow engine consumes this self-declared information to build the execution graph, eliminating the need for manual causal relationship definitions while maintaining execution control.
Solution Approach 2:
The patent introduces an intermediary layer (the workflow engine) that automatically infers causal relationships from transformation function signatures and metadata. This intermediary translates high-level function definitions into executable workflow sequences without requiring users to explicitly define dependencies.
3Reliability
If workflow definitions are environment-specific (as in Foundry bound to Apache Spark), then optimization for that environment can be achieved, but adaptability to other environments decreases
Solution Approach 1:
The system defines transformations in a universal, environment-agnostic manner using standard data processing functions and type annotations. The workflow engine can execute these same transformation definitions across multiple environments (local development, cloud platforms, different computing frameworks) without requiring environment-specific workflow definitions or bindings.
Solution Approach 2:
The patent segments the workflow definition into environment-independent transformation logic and environment-specific execution configuration. The core transformation functions are defined once in a universal format, and the workflow engine handles environment-specific adaptations automatically, allowing the same transformations to run in diverse environments.
4Reliability
If complex workflows are defined with explicit step ordering, then execution control is improved, but the difficulty of detecting and measuring increases and makes debugging difficult
Solution Approach 1:
The workflow engine acts as an intermediary that automatically builds the execution graph from transformation function metadata. It detects dependencies, determines execution order, and generates visualizations of the workflow structure, making the implicit dependencies explicit and easier to debug without losing execution control.
Solution Approach 2:
The system provides feedback mechanisms including automatic workflow visualization, dependency graph generation, and execution tracing. These feedback tools allow users to inspect the automatically inferred workflow structure, verify dependency relationships, and debug issues by observing actual execution paths without manually tracing through complex step definitions.
Data Source
AI summary
The present invention concerns the field of data engineering and analytics, and in particular the efficient creating and operating of data engineering workflows. A computer-implemented method is provided which comprises obtaining data defining a plurality of data transformations (200), and automatically generating a data transformation graph (100) based on the data transformations (200). Each data transformation (200) defines a step function (106) and at least one of a set of input datasets (104, 108) and a set of output datasets (108, 110), The data transformation graph (100) links the data transformations (200) by way of their input datasets (104, 108) and output datasets.