Distributed Data Processing Reusing Stored Results

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional distributed processing systems waste previous results by processing entire data sets every time, leading to inefficiencies and increased workloads on nodes.

Innovation Solution

A data processing method that splits data into segments, with previously processed segments being reused, allowing nodes to process only new segments in parallel, and performing secondary processing on stored results.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If the entire data set is processed every time data is entered, then data processing completeness is ensured, but processing efficiency deteriorates due to redundant work

Engineering Contradiction:

Improvedata processing efficiencyVSAvoidreprocessing time

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The system performs preliminary data processing and stores the results in memory before new data arrives. When new data is entered, the system checks for existing processing results that can be reused, avoiding redundant processing of unchanged data segments.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The data set is divided into multiple segments, and the system identifies which segments have changed and which remain unchanged. Only the changed segments are reprocessed, while unchanged segments leverage previously stored results, reducing overall processing time.

Inventive Principle:

Principle #1Segmentation

2Speed

If automatic data split and parallel processing is performed, then processing speed is improved, but node workload increases due to processing entire data sets repeatedly

Engineering Contradiction:

Improvedata processing speedVSAvoidnode computational energy

Core Design Contradiction:

SpeedVSLoss of energy

Solution Approach 1:

Instead of processing the entire data set in parallel across all nodes, the system performs partial processing only on changed data segments. The parallel processing framework is utilized selectively, assigning only the necessary portions of work to nodes, thereby reducing energy consumption while maintaining speed benefits.

Inventive Principle:

Principle #16Partial or excessive action

3Productivity

If previous processing results are not reused, then data processing accuracy is maintained, but processing efficiency deteriorates due to wasted computation

Engineering Contradiction:

Improveprocessing throughputVSAvoidresult management complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The system implements a feedback mechanism where processing results are stored in memory and subsequently checked against new processing requests. This feedback loop enables the system to identify and reuse valid previous results, improving throughput while the added complexity of result management is offset by the significant reduction in redundant computation.

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS9910821B2Data processing method, distributed processing system, and program

Publication Date: 2018.03.06 FUJITSU LTD

AI summary

A storage device stores results of first data processing previously performed. A splitting unit splits, with reference to the storage device, data into a first segment for which the results stored in the storage device are usable and a plurality of second segments for which the results stored in the storage device are not usable. A control unit assigns the plurality of second segments to a plurality of nodes, and uses the plurality of nodes in parallel to perform the first data processing on the plurality of second segments. A control unit exercises control so as to perform second data processing on a previous result corresponding to the first segment, which is stored in the storage device, and results obtained from the plurality of second segments using the plurality of nodes.