Cluster Job Manager Automation for Storage System Reliability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In clustered storage systems, administrative tasks are complex and require manual intervention if a job fails on one storage system, reducing system robustness and complicating management, as jobs often need to complete even in the event of system failures.
Innovation Solution
A job manager (JM) executes on each storage system to schedule and execute jobs, ensuring completion and re-initiating them on recovered or alternate systems, using a replicated database to track job affinities and status, with a master JM monitoring system status and handling failures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual intervention is required for job failure management in clustered storage systems, then administrative control is maintained, but system robustness is reduced and management complexity increases
Solution Approach 1:
The system implements self-service through automated job management where the job manager autonomously monitors job execution, detects failures, and re-initiates jobs on alternate storage systems without requiring administrator intervention. This self-managing capability resolves the contradiction by improving reliability through automation while reducing management complexity by eliminating manual intervention requirements.
Solution Approach 2:
The job manager implements continuous feedback mechanisms by monitoring job execution status and system health metrics. When failures are detected, the system automatically responds by re-initiating jobs on alternate systems. This closed-loop feedback control improves reliability while reducing management complexity by replacing manual monitoring and intervention with automated feedback-driven management.
2Ease of operation
If automated job management is implemented across storage systems, then manual intervention is reduced, but system complexity increases
Solution Approach 1:
The job manager is designed as a universal component that can execute on any storage system within the cluster and manage jobs with different affinities. It provides multi-functional capabilities including job scheduling, failure detection, and automatic re-initiation on alternate systems. This universal design improves ease of operation by providing consistent automated management while controlling system complexity through standardized multi-purpose functionality.
Solution Approach 2:
The job manager acts as an intermediary between the storage systems and the jobs, abstracting the complexity of job management from the underlying systems. It mediates job execution, failure detection, and recovery operations, thereby improving ease of operation while containing system complexity within the intermediary layer rather than propagating it throughout the entire system.
3Reliability
If jobs are re-initiated on alternate storage systems upon failure, then job completion is ensured, but coordination complexity increases
Solution Approach 1:
The system implements preliminary action by pre-establishing job affinity information in the replicated database and pre-identifying alternate storage systems capable of executing specific jobs. When failures occur, the job manager can immediately re-initiate jobs on pre-qualified alternate systems without complex real-time coordination. This preliminary preparation ensures job completion while reducing coordination complexity by avoiding ad-hoc decision-making during failure events.
Solution Approach 2:
The job manager uses copying of job information and affinity data from the replicated database to enable alternate systems to execute jobs. Instead of complex inter-system coordination, the system copies relevant job information to alternate storage systems that have the capability to execute them, thereby ensuring job completion while minimizing coordination complexity through information replication rather than active coordination.
Data Source
AI summary
A system and method for management of jobs in the clustered environment is provided. Each node in the cluster executes a job manager that interfaces with a replicated database to enable cluster wide management of jobs within the cluster. Jobs are queued in the replicated database and retrieved by a job manager for execution. Each job manager ensures that jobs are processed through completion or, failing that, are re-queued on another storage system for execution.