Execution Control Program for Distributed Data Processing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

In distributed processing systems like Hadoop, executing external programs that use multiple inputs with different formats is challenging due to constraints on data processing, increased operating time, human-caused mistakes, and high risks associated with porting and redevelopment.

Innovation Solution

A computer-readable medium stores an execution control program that inputs multiple input files with different formats, generates intermediate files with a common join key, sorts data, and generates output files for a join program, allowing direct execution of external programs without porting or extensive redevelopment.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If external programs are executed in distributed processing systems using conventional methods (Hadoop Streaming or reduce-side join), then processing capability is provided, but constraints on execution increase, operating time increases, and human errors increase

Engineering Contradiction:

Improveprocessing capabilityVSAvoidexecution constraints

Core Design Contradiction:

ProductivityVSEase of operation

Solution Approach 1:

The patent segments the data processing workflow into distinct stages: data preparation stage (where intermediate files are generated with join keys), sorting stage (where data is sorted by join key), and external program execution stage (where the prepared data is fed to external programs). This segmentation allows each stage to be optimized independently, reducing execution constraints while maintaining processing capability.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent performs preliminary actions by pre-processing the input data before external program execution. Specifically, it generates intermediate files with join keys, sorts the data by join key, and prepares the data in the required format ahead of time. This preliminary preparation reduces the constraints on external program execution by ensuring data is ready in the correct format when the external programs run.

Inventive Principle:

Principle #10Preliminary action

2Adaptability or versatility

If external programs are ported or redeveloped to execute in distributed processing systems, then execution is enabled, but operating time increases and human errors increase

Engineering Contradiction:

Improveexecution enablementVSAvoidoperating time

Core Design Contradiction:

Adaptability or versatilityVSLoss of time

Solution Approach 1:

The patent introduces intermediate files as a mediator between the distributed processing system and external programs. These intermediate files contain pre-processed data with join keys that match the requirements of external programs. By using this intermediary format, external programs can execute without being ported or redeveloped, thus enabling execution while avoiding the time loss associated with porting activities.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Adaptability or versatility

If data is processed through multiple transformation steps (format conversion, sorting, grouping), then compatibility with external programs is improved, but operating time increases

Engineering Contradiction:

Improveformat compatibilityVSAvoidoperating time

Core Design Contradiction:

Adaptability or versatilityVSDuration of action of moving object

Solution Approach 1:

The patent performs format conversion, sorting, and grouping as preliminary actions in the data preparation stage. By completing these transformation steps before external program execution, the system ensures format compatibility is achieved upfront, allowing external programs to run directly on the prepared data without additional transformation time during execution.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent merges multiple transformation operations (format conversion, join key addition, sorting, grouping) into an integrated data preparation process. By combining these steps into a unified workflow that generates intermediate files, the system achieves format compatibility while optimizing the total operating time through efficient resource utilization and parallel processing where applicable.

Inventive Principle:

Principle #5Merging (Combining)

Data Source

PatentUS10095699B2Computer-readable recording medium, execution control method, and information processing apparatus

Publication Date: 2018.10.09 FUJITSU LTD

AI summary

A slave computer reads a plurality of input files that have different formats and generates, for each of the input files, an intermediate file that has added thereto, as a join key, data in a type of column that is common to the input files. The slave computer sorts data in each intermediate file in accordance with the join key. The slave computer extracts, from each intermediate file that has data sorted therein, data that has the format of each of the input files so as to generate a plurality of output files that have data sorted therein with respect to each of the input files. The slave computer inputs the plurality of output files generated to a join program that joins data.