MapReduce Implementation in On-Demand Code Execution Systems

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Traditional implementations of the MapReduce programming model require a dedicated framework or infrastructure to manage execution of map and reduce functions, leading to inefficiencies and increased resource usage, particularly when processing large data sets.

Innovation Solution

An on-demand code execution environment that utilizes a stream data processing system as an intermediary between map and reduce function executions, eliminating the need for a centralized coordinator and allowing for dynamic scaling and parallelization without dedicated worker nodes.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If a dedicated framework or infrastructure is used to manage MapReduce execution, then coordination and control of map and reduce functions is achieved, but device complexity and resource usage increase

Engineering Contradiction:

Improvecoordination controlVSAvoidinfrastructure complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent extracts the coordination function from a dedicated framework and relocates it to existing message queue components. The message queue system assumes the coordination role, eliminating the need for separate framework infrastructure while maintaining reliable coordination between map and reduce tasks through standard queue mechanisms.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The message queue system performs multiple functions: it serves as both the data transmission medium and the coordination framework. Existing message queue infrastructure is leveraged to handle task scheduling, result collection, and synchronization, eliminating the need for dedicated framework components and reducing overall system complexity.

Inventive Principle:

Principle #6Universality (Multi-functionality)

2Productivity

If a centralized coordinator is implemented to manage MapReduce tasks, then execution control is improved, but network traffic and coordination overhead increase

Engineering Contradiction:

Improveexecution controlVSAvoidnetwork traffic

Core Design Contradiction:

ProductivityVSLoss of energy

Solution Approach 1:

The message queue acts as an intermediary between map and reduce tasks, enabling decentralized coordination. Instead of a centralized coordinator exchanging extensive messages with each task, the message queue provides a publish-subscribe mechanism where tasks independently publish results and subscribe to relevant events, reducing coordination traffic while maintaining execution control.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Reliability

If dedicated worker nodes are used for MapReduce processing, then task execution reliability is improved, but resource utilization efficiency decreases

Engineering Contradiction:

Improvetask executionVSAvoidresource utilization

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The patent merges MapReduce task execution with existing message queue consumer processes. Instead of dedicated worker nodes, the same infrastructure that consumes messages from queues is utilized to execute map and reduce tasks. This consolidation eliminates redundant infrastructure while maintaining reliable task execution through the proven message queue delivery guarantees.

Inventive Principle:

Principle #5Merging (Combining)

4Quantity of substance

If traditional MapReduce framework is used to process large data sets, then data processing capability is achieved, but processing time and resource consumption increase

Engineering Contradiction:

Improvedata processing volumeVSAvoidprocessing time

Core Design Contradiction:

Quantity of substanceVSLoss of time

Solution Approach 1:

The system performs preliminary actions by pre-configuring message queue subscriptions and routes before MapReduce tasks execute. Data flow paths are established in advance through queue topic configurations, allowing tasks to immediately process data without framework initialization overhead, thereby reducing processing time for large data volumes.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS11243953B2Mapreduce implementation in an on-demand network code execution system and stream data processing system

Publication Date: 2022.02.08 AMAZON TECH INC

AI summary

Systems and methods are described for providing an implementation of the MapReduce programming model utilizing tasks executing on an on-demand code execution system, utilizing a stream data processing system as an intermediary between map and reduce function. A map task implementing a map function can process portions of a data set, to generate outputs associated with different values for a measured attribute of the data set. Executions of the map task can publish outputs to a data stream on the stream data processing system, which stream is configured to utilize the measured attribute as a partition key for the stream. Based on the partition key, the stream data processing system can divide the stream into sub-streams, each containing a relevant subset of the outputs. The on-demand code execution system can execute a reduce task to apply the reduce function to the outputs of each sub-stream, thereby completing the MapReduce process.