Stream Processing Delay Compensation via Data Flow Rearrangement
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distribution stream processing, the variation in arrival times of data flows due to network delays and differing loads at edge bases leads to increased memory occupancy times, as data must be stored until all delayed data arrives, resulting in inefficient processing.
Innovation Solution
The information processing apparatus detects data flows with significant arrival delay differences and rearranges their generation elements to locations where delays are minimized, thereby reducing the variation in arrival times and suppressing memory occupancy times by optimizing data flow processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data flows are processed at multiple edge bases and merged, then processing capacity is improved, but variation in arrival times increases causing memory occupancy time to increase
Solution Approach 1:
The system performs preliminary detection of arrival delay differences before merging data flows. By identifying data flows with significant delay variations in advance, the system can prepare appropriate processing strategies, such as selecting whether to apply delay compensation or rearrangement operations, thereby preventing excessive memory occupancy before it occurs.
Solution Approach 2:
The system changes the processing parameters of data flows by detecting arrival delay differences and dynamically adjusting processing strategies. When delay variation exceeds a threshold, the system applies delay compensation techniques or rearranges processing order, effectively changing how data flows are handled to reduce memory occupancy time while maintaining processing capacity.
2Reliability
If data is stored until all delayed data arrives, then data completeness is improved, but memory occupancy time increases
Solution Approach 1:
The system implements feedback by continuously monitoring arrival delay differences of data flows and using this information to dynamically adjust processing decisions. The detection mechanism provides feedback about delay variations, which then triggers appropriate responses such as delay compensation or processing rearrangement, ensuring data completeness is maintained while minimizing unnecessary memory occupancy.
Solution Approach 2:
Instead of storing all data flows until the latest arrival (excessive action), the system applies partial action by selectively processing data flows based on their delay characteristics. Data flows with acceptable delay variations are processed without excessive waiting, while only those exceeding the threshold trigger delay compensation, thereby maintaining data completeness where needed while reducing overall memory occupancy time.
Data Source
AI summary
A non-transitory computer-readable recording medium has stored therein a program that causes a computer to execute a process, the process comprising: detecting a target data flow in a data flow group when receiving the data flow group and performing a merging process of the data flow group, the data flow group including a plurality of data flows processed at respective bases, the target data flow having a delay time that satisfies a predetermined condition; and executing rearrangement of a generation element of the target data flow to an environment such that differences between delay times of the plurality of data flows are reduced.