MapReduce Checkpointing for Spot Market Termination

Solution Overview

Problem

Conventional MapReduce implementations are ill-suited for spot market environments, where massive simultaneous machine terminations due to price fluctuations can lead to data loss and prolonged computation times, as they are designed to handle infrequent or smaller-scale failures rather than simultaneous terminations.

Innovation Solution

A distributed parallel processing framework that dynamically generates system checkpoints and stores completed portions of map tasks in persistent storage, allowing for seamless interruption and resumption of processing during system terminations, even in cloud computing infrastructures subject to spot market fluctuations.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If conventional MapReduce implementations are used in spot market environments, then processing can be performed using standard frameworks, but data loss occurs and computation times are prolonged during massive simultaneous machine terminations

Engineering Contradiction:

Improvedata preservation during terminationVSAvoidcomputation time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system performs preliminary actions by detecting impending terminations before they occur and proactively saving processing state to persistent storage. The master node monitors for termination signals and initiates state preservation procedures in advance, ensuring that computation progress is captured before machines are terminated, thus preventing data loss and enabling seamless resumption.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system implements beforehand cushioning by maintaining processing state in persistent storage as a buffer against termination events. This cushioning mechanism ensures that even when massive simultaneous terminations occur, the preserved state acts as a protective layer that prevents data loss and allows rapid recovery without prolonged computation times.

Inventive Principle:

Principle #11Beforehand cushioning (Prior cushioning)

2Productivity

If spot market pricing is utilized to reduce costs, then service rates can be optimized during low demand, but massive simultaneous machine terminations occur during high demand periods

Engineering Contradiction:

Improvecost optimizationVSAvoidprocessing continuity

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The system performs preliminary actions by detecting impending terminations before they occur and proactively saving processing state to persistent storage. The master node monitors for termination signals and initiates state preservation procedures in advance, ensuring that computation progress is captured before machines are terminated, thus preventing data loss and enabling seamless resumption.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system converts the harmful effect of spot market terminations into a benefit by using persistent storage to preserve processing state. Instead of allowing terminations to cause data loss and computation delays, the system leverages the termination events as opportunities to demonstrate the value of state preservation, enabling cost-effective spot market utilization while maintaining processing continuity through rapid recovery.

Inventive Principle:

Principle #22Blessing in disguise (Convert harm into benefit)

3Device complexity

If processing state is not preserved during terminations, then system complexity remains low, but data loss occurs and reprocessing is required

Engineering Contradiction:

Improvesystem complexityVSAvoidprocessing state

Core Design Contradiction:

Device complexityVSLoss of information

Solution Approach 1:

The system performs preliminary actions by detecting impending terminations before they occur and proactively saving processing state to persistent storage. The master node monitors for termination signals and initiates state preservation procedures in advance, ensuring that computation progress is captured before machines are terminated, thus preventing data loss and enabling seamless resumption.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentEP2535810B1System and method for performing distributed parallel processing tasks in a spot market

Publication Date: 2019.03.13 ACCENTURE GLOBAL SERVICES LTD

Figure 1
Figure 2
Figure 3

AI summary

As a result of the systems and methods described herein, an alternative MapReduce implementation is provided which monitors for impending termination notices, and allows dynamic checkpointing and storing of processed portions of a map task, such that any processing which is interrupted by large scale terminations of a plurality of computing devices - such as those resulting from spot market rate fluctuations - is preserved.