Adaptive Work Distribution via Status Object Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed systems waste resources by requiring partially completed data processing tasks to restart from the beginning, leading to inefficiencies in task distribution and re-distribution, especially for large or long-duration tasks.
Innovation Solution
Implementing a status object with start, current, and end markers to track task progress, allowing for the reassignment of tasks to resume processing from the point of completion, rather than from the beginning, and using a lease object to maintain task liveliness and efficiency in task distribution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If partially done tasks are restarted from the beginning, then task reliability is improved, but resource waste and processing time increase
Solution Approach 1:
The system performs preliminary tracking of task progress through status objects that record completion markers before failures occur. When a worker fails, the assignor queries the status object to retrieve the last known good progress point, allowing the task to resume rather than restart, thus avoiding waste of previously completed processing work
Solution Approach 2:
The system implements feedback mechanisms where workers periodically update status objects with their current progress on tasks. The assignor queries these status objects to monitor task completion state and make informed decisions about task reassignment, creating a closed-loop control system that adapts to actual task progress
2Device complexity
If simple task reassignment is used, then system complexity is reduced, but task progress tracking precision deteriorates
Solution Approach 1:
The system segments task information into distinct components: status objects that track progress, lease objects that manage task assignment state, and completion markers that record achievement points. This segmentation allows precise tracking of task progress while keeping each component's structure simple and manageable
Solution Approach 2:
The status object serves as an intermediary between the worker and the assignor, carrying progress information without requiring complex direct communication protocols. The lease object similarly mediates task assignment state, simplifying the interaction while enabling precise tracking of task lifecycle events
3Measurement precision
If frequent task status queries are performed, then task monitoring accuracy is improved, but system overhead and processing time increase
Solution Approach 1:
The system uses periodic polling where the assignor queries the status object at regular intervals or upon specific events (such as worker failure). This periodic approach balances monitoring accuracy with system overhead, avoiding continuous querying while ensuring timely detection of task completion or failure states
Data Source
AI summary
These methods, systems, and other aspects can provide for adaptive distribution/re-distribution of data processing tasks. An example method may create, by a processor in a first node, a status object for a data processing task and assign, by the processor in the first node, the data processing task to a worker of a second node. Further, after a first period of time, the method may query, by the processor in the first node, the status object for a task status of the assigned data processing task reported by the second node. The task status reflects a status of execution of the assigned data processing task by the second node. The method determines, by the processor in the first node, to reassign the assigned data processing task based on a received task status reported by the second node, and reassign, by the processor in the first node, the assigned task.