Graphical Data Pipeline Engine with Modular Operator Nodes

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Traditional data processing applications are inadequate for handling exceptionally voluminous and complex 'big data' sets, requiring advanced capabilities for ingestion, cleansing, storage, analysis, sharing, transformation, and visualization.

Innovation Solution

A data processing pipeline system that generates a user interface for clients to create and edit a graph representing a data processing pipeline by adding and interconnecting operator nodes with directed edges, allowing customization of nodes and indicating compatible data types through color and icon consistency, ultimately forming the basis for executing data processing operations on big data sets.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If traditional data processing applications are used, then the system structure is simple, but the capability to handle voluminous and complex big data sets is inadequate

Engineering Contradiction:

Improvecapability to handle big dataVSAvoidsystem structure

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent segments the data processing system into multiple operator nodes (e.g., ingestion, cleansing, storage, analysis, transformation operators) that can be independently selected and configured. Each operator handles a specific data processing function, allowing the system to build complex big data processing capabilities through composition of simple, modular components rather than requiring a monolithic complex system.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent creates a universal data processing pipeline system that can handle multiple types of big data operations (ingestion, cleansing, storage, analysis, transformation) through a common framework. The operator nodes are designed to be multi-functional and can be interconnected in various configurations to address different big data processing needs, making the system adaptable to diverse data processing requirements.

Inventive Principle:

Principle #6Universality (Multi-functionality)

2Ease of operation

If a graphical user interface is provided for creating data processing pipelines, then the ease of operation is improved, but the device complexity increases

Engineering Contradiction:

Improveease of creating pipelineVSAvoidinterface structure

Core Design Contradiction:

Ease of operationVSDevice complexity

Solution Approach 1:

The patent uses visual metaphors where operator nodes are represented as graphical icons or cards that users can select and manipulate. The graphical user interface copies the logical structure of the data processing pipeline into a visual representation, allowing users to interact with abstract data processing concepts through concrete visual elements. This makes the system easier to operate by translating complex pipeline configurations into intuitive graphical interactions.

Inventive Principle:

Principle #26Copying

Solution Approach 2:

The graphical user interface acts as an intermediary layer between the user and the underlying complex data processing system. Instead of requiring users to directly program or configure complex pipeline logic, the GUI provides a simplified interaction model where users can visually assemble pipelines by connecting operator nodes. This intermediary interface shields users from system complexity while enabling effective operation.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Measurement precision

If operator nodes are interconnected with directed edges to represent data flow, then the measurement precision of data flow is improved, but the device complexity increases

Engineering Contradiction:

Improvedata flow trackingVSAvoidgraph structure

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent employs visual indicators such as color coding and icons on operator nodes and connection edges to represent different data types and flow characteristics. This visual encoding system allows precise tracking of data flow through the pipeline without requiring complex textual annotations or configurations. Users can quickly understand data flow relationships and types through visual cues, achieving high measurement precision while maintaining interface simplicity.

Inventive Principle:

Principle #32Color changes

Data Source

PatentUS11275485B2Data processing pipeline engine

Publication Date: 2022.03.15 SAP SE

AI summary

A method for generating a data processing pipeline is provided. The method may include generating a user interface for displaying, at a client, a first operator node and a second operator node. The first operator node and the second operator node may each correspond to a data processing operation. In response to one or more inputs received from the client via the user interface, the first operator node and/or the second operator node may be added to a graph displayed in the user interface. The graph may be representative of a data processing pipeline. The first operator node and the second operator node may further be interconnected with an directed edge. The data processing pipeline may be generated based on the graph. Related systems and articles of manufacture, including computer program products, are also provided.