Priority-Aware Sampling Device for Parallel LDA Workload Balancing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The workload becomes unbalanced among units in parallel processing of Latent Dirichlet Allocation (LDA) due to fixed order sampling and model update methods, and existing solutions are inefficient for devices with a large number of units, making even data partitioning difficult.
Innovation Solution
A priority-aware sampling device dynamically assigns higher priorities to units with more unprocessed word tokens, allowing units with higher priorities to sample and update the model in parallel, thereby balancing workload and reducing model update conflicts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If static scheduling is used to resolve model update conflicts, then model update conflicts are avoided, but workload becomes unbalanced among units
Solution Approach 1:
The patent implements dynamic scheduling where the scheduler continuously monitors the number of unprocessed tokens in each unit's data subset and adjusts sampling priorities in real-time. Units with more unprocessed tokens receive higher priorities, allowing the system to adapt to changing workload conditions and maintain balance throughout the learning process.
Solution Approach 2:
The system incorporates feedback mechanisms where the scheduler receives information about each unit's current workload state (number of unprocessed tokens) and uses this feedback to dynamically adjust sampling priorities. This closed-loop control ensures that workload imbalances are detected and corrected through priority adjustments.
2Productivity
If data is partitioned into many subsets for parallel processing, then processing capacity is increased, but even partitioning becomes difficult
Solution Approach 1:
The patent performs preliminary data partitioning into P subsets at the beginning, where P is the vector length register size. This initial partitioning is done once before parallel processing begins, avoiding the need for complex dynamic partitioning during execution. The priority-based scheduling mechanism then handles the balancing without requiring re-partitioning.
3Ease of operation
If fixed order sampling is used, then implementation is simple, but workload balance deteriorates over time
Solution Approach 1:
The system transitions from static fixed-order sampling to dynamic priority-based sampling. The scheduler continuously updates sampling priorities based on current workload states, allowing units with more unprocessed tokens to be sampled more frequently. This dynamic adjustment maintains workload balance without significantly complicating the implementation.
Data Source
AI summary
A sampling device includes a priority assignment method that assigns higher priority to units with more remaining workload, a priority-aware scheduling method that enables units with higher priority to do the sampling and model update when a conflict happens, a modified priority-aware scheduling method that reduces scheduling overhead by re-assigning priority every several iterations, and another modified priority-aware scheduling method that explores different priority re-assignment frequencies and stores the sorted sequences in memory.