Resurrecting User Jobs in Compute Containers for Persistent Availability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Container services in a container environment face challenges in maintaining persistent availability and resilience, particularly when a container becomes corrupted or fails, leading to loss of intermediary outputs and difficulties in backup and recovery, especially in hybrid cloud contexts.
Innovation Solution
A method is implemented to ensure persistent availability of container services by using loosely coupled concurrency and redundancy, where a computing device assigns user jobs to compute runner agents, monitors progress, stores artifacts, and resurrects user jobs in new compute container instances upon failure, ensuring data replication and recovery across multiple sessions and load balancers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a container is used to run a user job, then the user job can be executed in an isolated environment with bundled software resources, but the container may become corrupted or fail, leading to loss of intermediary outputs and unavailability of services
Solution Approach 1:
The system performs preliminary actions by creating backup container instances before the primary container fails. The backup instances are pre-configured with the same software resources and are ready to take over immediately when corruption or failure is detected, preventing service interruption.
Solution Approach 2:
When a container becomes corrupted or fails, the system discards the failed container instance and recovers by resurrecting the user job in a newly created container instance. The intermediary outputs are preserved and transferred to the new container, ensuring continuity of service.
2Reliability
If container instances are created for each user job, then isolation and resource bundling are achieved, but backup and recovery become difficult when containers fail or are corrupted
Solution Approach 1:
The system creates copy instances of container configurations for backup purposes. When a container fails, the system copies the necessary software resources and job state from the failed container to a new container instance, enabling rapid recovery without manual intervention.
Solution Approach 2:
Backup container instances are created in advance with all necessary software resources bundled. This preliminary preparation ensures that when recovery is needed, the system can quickly provision a new container with pre-configured resources rather than rebuilding from scratch.
3Productivity
If a runner agent is assigned to execute user jobs, then job execution is automated and monitored, but the system becomes vulnerable to runner agent failures that terminate user jobs before completion
Solution Approach 1:
The system creates backup runner agents in advance that are ready to take over if the primary runner agent fails. These backup agents are pre-configured with the necessary credentials and monitoring capabilities to immediately assume control of user jobs without interruption.
Solution Approach 2:
The system implements continuous monitoring of runner agent health and job execution status. When a runner agent failure is detected, the feedback mechanism triggers automatic resurrection of the user job with a new runner agent, ensuring job completion reliability while maintaining automated execution.
Data Source
AI summary
A method makes container services persistently available. A computing device receives a request for implementation of a user job in a container environment, and assigns the user job to a compute runner agent of a plurality of compute runner agents to execute the user job. Each compute runner agent is associated with a compute container instance having a unique compute container identifier corresponding to the user job. A computing device assigns the user job to a balancer task to monitor progress of the user job, and assigns the user job to a storage agent to store artifacts associated with running the user job. A computing device receives a notification from the balancer task describing whether the runner agent is correctly running the user job. In response to the runner agent incorrectly running the user job, a computing device resurrects the user job in a new compute container instance.