Dynamic Data Pipeline Rule Switching via Trigger Signals

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional data pipelines require system downtime for configuration changes, disrupting data processing and leading to data backlogs and system instability due to the need to maintain processing order across components.

Innovation Solution

A method where a data processing system uses trigger data to dynamically change reading rules for data pipelines, allowing new data pipes to be created in parallel with existing ones, enabling continuous data processing without stopping the system by shifting data processing between data pipes based on operational states and processing times.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If the configuration of the data pipeline is changed by reactivating the system, then the data pipeline configuration can be updated, but the system must be temporarily stopped causing downtime and disruption

Engineering Contradiction:

Improvedata pipeline configuration change capabilityVSAvoidsystem downtime

Core Design Contradiction:

Adaptability or versatilityVSLoss of time

Solution Approach 1:

The patent implements dynamic configuration changes by allowing the data pipeline to switch between multiple configuration versions without stopping. The system maintains current and new configuration versions simultaneously, enabling seamless transitions that eliminate system downtime while updating the data pipeline configuration.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The patent prepares new configuration versions in advance before they are needed. By pre-loading and validating new configuration versions while the system operates with the current configuration, the system can switch configurations instantly without interruption, avoiding downtime during configuration updates.

Inventive Principle:

Principle #10Preliminary action

2Adaptability or versatility

If the data pipeline configuration is changed, then the system can adapt to new requirements, but the processing order in components may be disrupted causing data backlogs

Engineering Contradiction:

Improveconfiguration adaptabilityVSAvoiddata processing order maintenance

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The patent implements feedback mechanisms that monitor data flow and processing status during configuration transitions. This feedback system detects potential disruptions to processing order and adjusts the configuration switching process to maintain data integrity and prevent backlogs, ensuring reliable operation during adaptability changes.

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The patent prepares compensatory measures in advance by maintaining buffer capacities and validation mechanisms before configuration changes occur. These pre-prepared cushioning measures prevent data backlogs and processing order disruptions by absorbing transient imbalances that may occur during configuration transitions.

Inventive Principle:

Principle #11Beforehand cushioning (Prior cushioning)

Data Source

PatentUS10409740B2System for changing rules for data pipleine reading using trigger data from one or more data connection modules

Publication Date: 2019.09.10 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

Provided is an operating method of a system including a first data processing module, a second data processing module, and one or more data connection modules that transfer data between the first data processing module and the second data processing module. In an operational state where the first data processing module and the second data processing module can process data input thereto, the first data processing module inputs trigger data to any of the one or more data connection modules, and in response to the second data processing module receiving the trigger data via the one or more data connection modules, the second data processing module changes a reading rule for reading data from the one or more data connection modules. Also provided are a system and a computer program product.