Stream Processing Task Reordering to Reduce Data Reintroduction Load

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Stream processing systems face increased processing load and time due to data reintroduction against the stream, which is not efficiently managed, especially when the amount of data to be reintroduced increases, and it is challenging to optimize the processing load in real-time changing environments.

Innovation Solution

An information processing method that determines the arrangement order of tasks from upstream to downstream in a stream processing system by acquiring patterns of data reintroduction, specifying the amount of data to be reintroduced between tasks, and selecting a pattern with a relatively small processing load, thereby reducing the overall processing load and time.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If data is reintroduced from a downstream task to an upstream task in a stream processing system, then the data can be reused by upstream tasks, but the processing load and time increase significantly

Engineering Contradiction:

Improvedata reuse capabilityVSAvoidprocessing load

Core Design Contradiction:

Adaptability or versatilityVSProductivity

Solution Approach 1:

The system performs preliminary analysis of data reintroduction patterns and task arrangement options before executing the stream processing. By evaluating multiple candidate arrangements in advance and selecting the optimal one, the system prepares the processing pipeline to minimize subsequent processing load while enabling necessary data reuse.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system dynamically adjusts the task arrangement based on the actual data reintroduction patterns observed during processing. By flexibly reconfiguring the stream processing topology according to real-time conditions, the system optimizes the balance between data reuse and processing load.

Inventive Principle:

Principle #15Dynamics

2Adaptability or versatility

If data is reintroduced from a downstream task to an upstream task, then data reuse is enabled, but the processing time increases

Engineering Contradiction:

Improvedata reuse capabilityVSAvoidprocessing time

Core Design Contradiction:

Adaptability or versatilityVSLoss of time

Solution Approach 1:

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system identifies and prioritizes critical data flow paths that require reintroduction, allowing these essential operations to proceed with minimal delay while optimizing or skipping less critical processing steps to reduce overall processing time.

Inventive Principle:

Principle #21Skipping (Rushing through)

3Quantity of substance

If the amount of data to be reintroduced increases, then more data can be reused by upstream tasks, but the processing load increases significantly

Engineering Contradiction:

Improveamount of data reusedVSAvoidprocessing load

Core Design Contradiction:

Quantity of substanceVSProductivity

Solution Approach 1:

The system applies different processing strategies to different portions of the data stream based on local characteristics. By identifying specific segments or types of data that benefit most from reintroduction and optimizing their handling separately, the system maximizes data reuse value while minimizing the overall processing load impact.

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The system changes processing parameters such as batch size, parallelism degree, or compression levels based on the amount of data requiring reintroduction. By dynamically adjusting these parameters, the system handles larger data volumes with reintroduction more efficiently, reducing the processing load increase.

Inventive Principle:

Principle #35Parameter changes

4Productivity

If multiple patterns of task arrangement are evaluated, then the optimal arrangement can be selected, but the system complexity increases

Engineering Contradiction:

Improveprocessing efficiencyVSAvoidsystem complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The system divides the task arrangement evaluation into discrete, manageable patterns or configurations. By segmenting the complex optimization problem into evaluating a finite set of candidate arrangements, the system can systematically compare options and select the optimal one without being overwhelmed by the full complexity of all possible permutations.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system uses simplified models or approximations for evaluating task arrangement patterns rather than performing exhaustive detailed analysis. By using lightweight evaluation metrics and disposable candidate configurations that can be quickly assessed and discarded, the system reduces computational complexity while still identifying effective arrangements.

Inventive Principle:

Principle #27Cheap short-living objects (Disposable)

Data Source

PatentUS11599388B2Processing program to rearrange order of task in stream processing

Publication Date: 2023.03.07 FUJITSU LTD

AI summary

An information processing method for determining a pattern that indicates an arrangement order of the plurality of tasks from upstream to downstream of a stream is performed by a computer. The method includes acquiring a plurality of patterns to be candidates of an arrangement order of the plurality of tasks from upstream to downstream of the stream in a case of executing the plurality of tasks using a stream processing format; specifying, for each pattern of the plurality of acquired patterns, an amount of data to be reintroduced from one task of the plurality of tasks to another task located upstream side of the stream with respect to the one task; and determining the pattern from among the plurality of patterns based on the specified amount of data to be reintroduced for the each pattern.