Cloud Job Replay System for Failover Synchronization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transitioning operations to a new service after a catastrophic failure in web-based services is expensive and time-consuming due to the need to resubmit jobs and maintain synchronization between primary and secondary locations.
Innovation Solution
A cloud management system that replays jobs submitted to a primary location at a secondary location using a mapping of job input parameters, allowing jobs to determine necessary changes and resubmit or remain unchanged based on fail-over events, with features like idempotent APIs and asynchronous processing to handle intermittent network failures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If jobs are resubmitted to a secondary location after a fail-over event, then service continuity is maintained, but the complexity of job management and synchronization increases
Solution Approach 1:
The patent creates a copy of the primary service location as a secondary location, which can take over when the primary fails. Jobs are copied from the primary location to the secondary location, allowing the secondary to process identical workloads without requiring complex custom integration logic. This copying approach maintains service continuity while keeping the failover mechanism relatively simple.
Solution Approach 2:
The patent introduces a failover management system that acts as an intermediary between the primary and secondary service locations. This intermediary handles the complex tasks of job tracking, synchronization, and coordinate updates, isolating the complexity from the core service logic. The intermediary maintains a mapping of job coordinates and manages the resubmission process automatically.
2Reliability
If comprehensive job tracking and resubmission is implemented, then no work is lost during failover, but the time and resources required for synchronization increase
Solution Approach 1:
The patent implements preliminary tracking of job coordinates and status information before failover occurs. The system maintains a mapping of submitted jobs and their current state in advance, so when failover is needed, the resubmission can proceed immediately using pre-collected information. This eliminates the need for time-consuming discovery and coordination during the actual failover event.
Solution Approach 2:
The patent employs feedback mechanisms where the secondary service location receives information about job submission status and coordinates from the primary location. This feedback loop allows the system to understand which jobs need resubmission and their current state, enabling efficient recovery without redundant operations. The feedback ensures that already-completed jobs are not resubmitted unnecessarily.
3Reliability
If a secondary service location is maintained for failover, then service availability improves, but the cost of maintaining duplicate infrastructure increases
Solution Approach 1:
The patent merges the functionality of the primary and secondary service locations by having them process the same type of jobs with the same service logic. Rather than maintaining completely separate systems, the secondary location is integrated into the same service architecture, allowing shared resources and coordinated operation. This reducing redundancy while maintaining failover capability.
Solution Approach 2:
The patent creates a secondary service location that copies the essential functionality and service logic of the primary location. This copying approach ensures the secondary can independently handle workloads when needed, while requiring less infrastructure than a fully redundant system. The secondary location maintains sufficient duplication for failover but can operate with reduced resources during normal conditions.
Data Source
AI summary
Jobs submitted to a primary location of a service within a period of time before and/or after a fail-over event are determined and are resubmitted to a secondary location of the service. For example, jobs that are submitted fifteen minutes before the fail-over event and jobs that are submitted to the primary network before the fail-over to the second location is completed are resubmitted at the secondary location. After the fail-over event occurs, the jobs are updated with the secondary network that is taking the place of the primary location of the service. A mapping of job input parameters (e.g. identifiers and/or secrets) from the primary location to the secondary location are used by the jobs when they are resubmitted to the secondary location. Each job determines what changes are to be made to the job request based on the job being resubmitted.