Task Orchestration System Probing Hidden Schedules
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Computing systems face challenges in scheduling tasks due to hidden or decentralized control of prerequisite completion times, leading to task failures and inefficiencies in processing.
Innovation Solution
A method for monitoring upstream computations by constructing a test task schedule to probe the hidden schedule of a remote computing system, estimating prerequisite processing time, and generating a main task schedule to optimize task success rates, using techniques such as binary search algorithms and machine learning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If tasks are submitted to a remote computing system with hidden prerequisite schedules, then task throughput can be increased, but task failure rate increases due to unknown prerequisite completion timing
Solution Approach 1:
The system performs preliminary actions by submitting test batches before the main task batch to probe the hidden prerequisite schedule. These test batches are submitted at different time offsets to discover when prerequisites become available, allowing the system to determine optimal submission timing before committing the main batch.
Solution Approach 2:
The system implements feedback by monitoring the success or failure of test batch submissions and using this information to adjust the timing of main batch submissions. The feedback loop continues by observing prerequisite completion patterns and refining future submission schedules to maximize success rates.
2Ease of operation
If task submission timing is randomized or evenly distributed, then scheduling simplicity is maintained, but task failure occurs due to misalignment with hidden prerequisite schedules
Solution Approach 1:
The system applies partial action by submitting a small subset of test batches with exaggerated time offsets from the main batch. These test batches use excessive timing variations to probe the hidden schedule boundaries, while the main batch uses the optimized timing derived from test results.
Solution Approach 2:
Test batches serve as intermediaries between the task submission system and the hidden prerequisite schedule. These intermediary batches absorb the complexity of schedule discovery, allowing the main batch to be submitted at optimized times without directly encountering the hidden schedule uncertainties.
3Reliability
If test batches are submitted to probe hidden schedules, then task success rate improves, but system resource overhead increases due to additional probing tasks
Solution Approach 1:
The system changes parameters by varying the time offset between test batches and main batches across different iterations. By adjusting the temporal parameter of batch submission, the system efficiently probes the hidden schedule with minimal batches, reducing the overhead of schedule discovery.
Solution Approach 2:
The system skips unnecessary probing iterations by using binary search or exponential backoff strategies to rapidly converge on the optimal submission timing. Instead of linearly testing every possible time offset, the system jumps through the time space efficiently, reducing the number of test batches needed.
Data Source
AI summary
An example computer-implemented method is provided that includes: determining a remote computing system associated with a task queue comprising a plurality of processing tasks, wherein the remote computing system updates, based on a hidden schedule, prerequisites for the plurality of processing tasks; initiating the plurality of processing tasks according to a test task schedule constructed to probe the hidden schedule of the remote computing system; receiving a plurality of responses respectively for the plurality of processing tasks, wherein a respective response corresponds to at least one of a success indicator or a failure indicator; estimating, based on the plurality of responses, a prerequisite processing time of the hidden schedule; and generating, based on the estimated prerequisite processing time, a main task schedule for the task queue.