Containerized Pipeline Checkpointing for Partial Re-execution

Solution Overview

Problem

Current containerized application pipelines require the entire pipeline to be re-executed even if only a portion fails, wasting resources and inefficiency due to lack of mid-point execution capability.

Innovation Solution

Implementing a system that stores task results and generates checkpoints to allow dynamic re-execution of pipelines from a specific point, reusing previous results to execute only the necessary parts of the pipeline.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If the entire pipeline is re-executed when a task fails, then the pipeline can be ensured to run correctly from scratch, but resource wastage increases and execution time increases

Engineering Contradiction:

Improvepipeline execution correctnessVSAvoidresource wastage

Core Design Contradiction:

ReliabilityVSLoss of energy

Solution Approach 1:

The pipeline is divided into discrete tasks with checkpoints between them. When a task fails, only the tasks after the checkpoint need re-execution, not the entire pipeline. This segmentation allows selective re-execution of only the necessary portions, reducing resource wastage while maintaining correctness.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Checkpoints are established in advance during pipeline execution to capture the state at specific points. These pre-established checkpoints enable rapid resumption of execution from the last known good state, avoiding the need to re-execute tasks that have already succeeded.

Inventive Principle:

Principle #10Preliminary action

2Reliability

If the entire pipeline is re-executed when a task fails, then pipeline correctness is maintained, but execution time increases

Engineering Contradiction:

Improvepipeline execution correctnessVSAvoidexecution time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The pipeline is divided into discrete tasks with checkpoints between them. When a task fails, only the tasks after the checkpoint need re-execution, not the entire pipeline. This segmentation allows selective re-execution of only the necessary portions, reducing resource wastage while maintaining correctness.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Checkpoints are established in advance during pipeline execution to capture the state at specific points. These pre-established checkpoints enable rapid resumption of execution from the last known good state, avoiding the need to re-execute tasks that have already succeeded.

Inventive Principle:

Principle #10Preliminary action

3Loss of energy

If checkpoints are implemented for partial re-execution, then resource wastage is reduced and execution time is improved, but system complexity increases

Engineering Contradiction:

Improveresource wastageVSAvoidsystem complexity

Core Design Contradiction:

Loss of energyVSDevice complexity

Solution Approach 1:

A checkpoint manager acts as an intermediary component that handles checkpoint creation, storage, and restoration. This dedicated mediator simplifies the overall system architecture by centralizing the complexity of checkpoint management, making the rest of the pipeline execution logic simpler and more maintainable.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS20240152371A1Dynamic re-execution of parts of a containerized application pipeline

Publication Date: 2024.05.09 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

A method including: storing results of a task in a pipeline executed in a container running in a computing environment; generating a check point for the task; and re-executing the pipeline from the check point for the task reusing the results.