Distributed Job Allocation via Deterministic Worker Sorting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In cloud-based storage systems, efficiently reallocating long-lived jobs among worker processes without relying on a central arbiter is challenging, especially when worker processes fail due to hardware and software errors, requiring a distributed technique to balance job distribution and minimize job reassignment time.
Innovation Solution
A distributed computing system where worker processes identify available peers by querying a distributed consensus service, sort themselves by unique identifiers, and use a deterministic pseudorandom number generator to evenly assign jobs, ensuring idempotent operations to prevent correctness issues during reassignment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If a central arbiter is used to perform reallocation operations, then job distribution can be controlled centrally, but system scalability and fault-tolerance are reduced
Solution Approach 1:
Each worker process independently performs job allocation by sorting itself and other workers by unique identifiers, then using local deterministic pseudorandom number generators to assign jobs. This self-service approach eliminates the need for a central arbiter while maintaining consistent and scalable operation across the distributed system.
Solution Approach 2:
The patent introduces a distributed consensus service as an intermediary that enables workers to discover available workers and coordinate job allocation without requiring direct central control. This mediator facilitates decentralized decision-making while maintaining system coherence.
2Reliability
If jobs are reallocated among worker processes, then fault-tolerance is improved, but job reassignment time increases
Solution Approach 1:
Workers perform preliminary sorting of themselves and other workers by unique identifiers before job assignment. This pre-computed ordering enables O(1) job assignment through deterministic pseudorandom number generation, significantly reducing reassignment time while maintaining fault-tolerance through continuous worker availability monitoring.
Solution Approach 2:
The patent changes the allocation parameter from centralized control to decentralized deterministic random assignment. Each worker uses its unique identifier to seed a deterministic pseudorandom number generator, creating reproducible yet distributed assignment decisions that minimize reassignment overhead.
3Loss of time
If the number of jobs moved during reallocation is minimized, then reassignment overhead is reduced, but allocation flexibility is limited
Solution Approach 1:
The patent replaces mechanical centralized allocation control with a mathematical approach using deterministic pseudorandom number generation. Each worker independently computes its job assignment based on sorted worker identifiers and local random number generation, achieving both low overhead and high flexibility through mathematical rather than mechanical control.
Data Source
AI summary
A distributed computing system that executes a set of long-lived jobs is described. During operation, each worker process performs the following operations. First, the worker process identifies a set of jobs to be executed and a set of worker processes that can execute the set of jobs. Next, the worker process sorts the set of worker processes based on unique identifiers for the worker processes. Then, the worker process assigns jobs to each worker process in the set of worker processes, wherein approximately the same number of jobs is assigned to each worker process, and jobs are assigned to the worker processes in sorted order. While assigning jobs, the worker process uses an identifier for each worker process to seed a pseudorandom number generator, and then uses the pseudorandom number generator to select jobs for each worker process to execute.