Data Analytics Pipeline Configuration via Directed Acyclic Graph

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional big data transformation engines require significant time and labor for writing boilerplate code, integrating disparate data sources and sinks, orchestrating tasks, and pipelining complex transformations, which diverts resources away from business logic and functional testing.

Innovation Solution

A method using a configuration file to generate a directed acyclic graph (DAG) that specifies data sources, transformations, and sinks, executed through an API, allowing for optimized data analytics task management without the need for extensive boilerplate code.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If conventional big data transformation engines are used, then data processing tasks can be performed, but significant time and labor resources are required for writing boilerplate code

Engineering Contradiction:

Improvedata processing efficiencyVSAvoidtime for writing boilerplate code

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The patent uses template-based code generation to create boilerplate code automatically. Pre-defined templates for common data processing patterns (reading data, transforming data, writing results) are copied and instantiated with specific parameters, eliminating the need to manually write repetitive boilerplate code while maintaining functionality.

Inventive Principle:

Principle #26Copying

Solution Approach 2:

The patent introduces a code generation intermediary layer between the user's high-level configuration and the actual execution engine. This intermediary automatically translates configuration files into executable code, mediating between user intent and system execution without requiring users to write low-level boilerplate code.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Productivity

If conventional approaches are used for integrating disparate data sources and orchestrating tasks, then data analytics can be performed, but the process becomes difficult and resource-intensive

Engineering Contradiction:

Improvedata analytics capabilityVSAvoidcomplexity of integrating data sources and orchestrating tasks

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent creates a universal configuration framework that handles multiple data sources (relational databases, NoSQL databases, file systems), multiple transformation types, and various sinks through a single unified configuration interface. This multi-functional approach eliminates the need for separate integration logic for each data source type, reducing complexity while maintaining versatility.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The patent uses parameterized configuration files where data sources, transformations, and sinks are defined through configurable parameters rather than hard-coded logic. By changing parameters in the configuration file, users can integrate different data sources and orchestrate various tasks without modifying the underlying system complexity, achieving flexibility through parameter management.

Inventive Principle:

Principle #35Parameter changes

3Reliability

If extensive unit testing is performed during development, then code reliability improves, but resources available for business logic development decrease

Engineering Contradiction:

Improvecode reliabilityVSAvoidbusiness logic development speed

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The patent implements self-service through automated code generation and configuration validation. The system automatically generates tested, reliable boilerplate code from validated configuration files, and performs configuration validation to ensure correctness before execution. This self-service approach to reliability reduces the need for manual unit testing while maintaining code quality.

Inventive Principle:

Principle #25Self-service

4Adaptability or versatility

If significant code changes are made to upgrade big data transformation engines, then new features are obtained, but development stability decreases

Engineering Contradiction:

Improveengine upgrade capabilityVSAvoiddevelopment stability

Core Design Contradiction:

Adaptability or versatilityVSStability of the object's composition

Solution Approach 1:

The patent segments the system into configuration layer and execution layer. The configuration file (layer 1) defines data sources, transformations, and sinks in a stable, version-controlled format. The code generation and execution (layer 2) can be upgraded independently without affecting the configuration. This segmentation allows engine upgrades while maintaining configuration stability and reducing the need for extensive code changes.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS10860297B2Methods for efficiently managing data analytics using complex dependency pipelines and devices thereof

Publication Date: 2020.12.08 JPMORGAN CHASE BANK NA

AI summary

Methods and systems for managing data analytics are provided. The methods and systems entail obtaining a configuration file that includes steps. Each step includes at least one parameter that identifies a source of input data, a transformation that corresponds to business logic to be applied to the input data, and/or a sink of output data resulting from a transformation to the input data. A directed acyclic graph is generated based on the configuration file. The directed acyclic graph is then executed by invoking an application programming interface (API) of a data transformation engine to apply the transformations to the input data, and then returning an indication of completion to a source of the configuration file.