Healthbus Mediator for Long-Running Workflow Continuity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Container orchestration systems face challenges in supporting long running workflows as service replicas can go offline, leading to abandoned or failed processes, especially since existing solutions only scale for synchronous services and lack information about the state of replicas for long running workflows.
Innovation Solution
Implementing a healthbus within the container orchestration system where each pod periodically publishes and queries health messages to maintain a world view of the system, allowing for the identification of inactive pods and ensuring only one replica continues long-running processes, thereby preventing duplicate execution and enabling optimal scaling and request management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If service replicas are deployed to handle workflows, then system scalability and availability are improved, but long running workflows may be lost or abandoned when replicas go offline
Solution Approach 1:
A healthbus is introduced as an intermediary communication mechanism between service replicas. Each replica periodically publishes health messages containing its identification and active time interval to the healthbus, and queries the healthbus to determine the system world view. This mediator enables replicas to share vital information about their state and workflow progress without direct peer-to-peer communication, ensuring workflow continuity even when replicas go offline.
2Loss of information
If replicas periodically publish health messages to maintain system awareness, then workflow state tracking is improved, but system complexity increases
Solution Approach 1:
Instead of continuous communication, replicas perform periodic publishing of health messages containing their identification and active time interval. This periodic action reduces communication overhead while maintaining sufficient system awareness. The healthbus collects these periodic updates and makes them available for queries, achieving replica state tracking without requiring constant system-wide communication.
3Difficulty of detecting and measuring
If the system tracks active time intervals of replicas, then detection of inactive replicas is improved, but measurement overhead increases
Solution Approach 1:
The system performs preliminary action by having replicas publish their active time interval information proactively in health messages before needing to detect inactivity. This preliminary data collection eliminates the need for complex active monitoring mechanisms. When a replica becomes inactive, its absence from expected health message publication or changes in its recorded active time interval immediately signals its inactive state, enabling efficient detection without continuous measurement overhead.
Data Source
AI summary
Described herein are systems and methods for sharing vitals among service replicas to enable processing of long running workflows within a container orchestration system. A method can provide a container orchestration system that provides within one or more container orchestration environments, a runtime for containerized workloads and services. The method can provide a healthbus within the container orchestration system, the healthbus comprising a memory. The method can deploy a plurality of pods within the container orchestration system, each pod comprising a memory. The method can periodically publish, by each pod, a health message to the healthbus, the health message comprising at least an indication of an identification of the pod and an indication of a time interval in which the pod has been active. The method can periodically query, by each pod, the healthbus to determine a world view of the container orchestration system.