Stream Data Processing With Asynchronous External Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Stream processing systems experience a decrease in processing speed when executing tasks that require access to external information processing systems, as these systems suspend task execution until the external access is completed, affecting both tasks that require external access and those that do not.
Innovation Solution
The system implements a data processing program that allows tasks to continue executing on new datasets without waiting for external access results, by canceling the execution of tasks that require external access, making the access asynchronously, and re-executing the task after receiving the access result.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the stream processing system suspends task execution until external access is completed, then the accuracy of data processing is improved, but the processing speed decreases
Solution Approach 1:
The patent segments task execution into two independent paths: one for tasks requiring external access and another for tasks that do not. By separating these paths, the system can maintain accurate processing for access-required tasks while continuing processing for other tasks, thus resolving the contradiction between accuracy and speed.
Solution Approach 2:
The patent dynamically adjusts task execution based on external access requirements. When external access is needed, the system pauses only the affected tasks while allowing other tasks to continue. This dynamic adaptation enables the system to maintain both accuracy (by completing necessary external accesses) and processing speed (by not blocking all tasks).
2Speed
If the system cancels task execution to make external access asynchronously, then the processing speed is improved, but the complexity of task management increases
Solution Approach 1:
The patent introduces a barrier marker as an intermediary mechanism to coordinate between tasks that require external access and tasks that do not. The barrier marker simplifies management by providing a clear synchronization point, reducing the complexity of coordinating asynchronous task execution across different task types.
Solution Approach 2:
The system uses feedback mechanisms through barrier markers to track the status of external accesses and coordinate task resumption. This feedback loop enables the system to manage complexity by providing clear, actionable information about task completion status, making the asynchronous management more controllable.
3Reliability
If the system waits for external access results before executing tasks, then the reliability of data processing is improved, but the time consumption increases
Solution Approach 1:
The patent performs preliminary actions by canceling task execution before external access is completed, allowing other tasks to proceed in the meantime. This preliminary cancellation of blocking operations enables the system to maintain reliability for critical tasks while reducing overall time consumption by not waiting for all tasks to complete sequentially.
Solution Approach 2:
The system maintains continuity of useful action by allowing tasks that do not require external access to continue executing without interruption. This continuous execution of non-blocking tasks reduces time consumption while the system ensures reliability for tasks that do require external access by coordinating their execution through barrier markers.
Data Source
AI summary
A non-transitory computer-readable storage medium storing a program that causes a computer to execute a process. The process includes, in response to reception of a first dataset, starting to execute a task on the first dataset, when execution of the task involves execution of a first process requesting an access to information processing system, cancelling the execution of the task, and making the access to the information processing system for the first process, in response to reception of a second dataset, starting to execute the task on the second dataset, continuing the execution of the task on the second dataset when the execution of the task on the second dataset does not involve execution of the first process, and when an access result for the access is received, re-executing the task on the first dataset after the execution of the task on the second dataset is completed.