Archive Manager for Map-Reduce Jobs

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Map-reduce frameworks face challenges in scaling to update a single file concurrently, making it difficult to output results into a portable archive file that can be easily read by various applications, as distributed file systems do not allow multiple tasks to lock and update a single file simultaneously.

Innovation Solution

An archive manager is introduced to handle calls from map-reduce jobs, using a buffering database as a temporary cache to buffer updates, allowing for reading and writing to an archive file while managing access to ensure only one job can write at a time, and serializing updates to create a single archive file.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If multiple map-reduce tasks concurrently update a single archive file, then productivity is improved, but reliability deteriorates due to file locking conflicts and data inconsistency

Engineering Contradiction:

Improveconcurrent update capabilityVSAvoiddata consistency

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The patent segments the single archive file update operation into multiple independent update files, each handled by a separate map-reduce task. This allows concurrent updates without locking conflicts, while maintaining data consistency through the merge operation that combines all update files into the final archive file.

Inventive Principle:

Principle #1Segmentation

2Ease of operation

If a single archive file is used for output, then ease of operation is improved, but device complexity increases due to locking mechanisms and access control

Engineering Contradiction:

Improvesingle file outputVSAvoidlocking mechanism

Core Design Contradiction:

Ease of operationVSDevice complexity

Solution Approach 1:

The patent extracts the locking mechanism and access control complexity from the archive file update process by using multiple independent update files. Each task writes to its own file without needing locks, and the complexity is eliminated rather than managed.

Inventive Principle:

Principle #2Taking out (Extraction)

3Productivity

If distributed file system is used for storage, then productivity is improved, but adaptability deteriorates for portable archive formats

Engineering Contradiction:

Improvedistributed processingVSAvoidportability

Core Design Contradiction:

ProductivityVSAdaptability or versatility

Solution Approach 1:

The patent combines the benefits of distributed processing with portable archive formats by having multiple tasks write to separate update files in the distributed file system, then merging all update files into a single portable archive file that can be easily transferred and read by various applications.

Inventive Principle:

Principle #5Merging (Combining)

Data Source

PatentUS11556496B2Outputting map-reduce jobs to an archive file

Publication Date: 2023.01.17 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

Method and system are provided for writing output from map-reduce jobs to an archive file. The method may include providing an archive manager and exposing an interface to be called from map-reduce jobs to output to an archive file in a map-reduce distributed file system. The method may also include using a buffering database as a temporary cache to buffer updates to the archive file. Handling by the archive manager calls from map-reduce jobs may allow: reading directly from an archive file or from a job index at the buffering database; writing to a job index at the buffering database used as a temporary cache to buffer updates; and serializing updates from the buffering database to the archive file.