Skew Exception Detection in Parallel Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing techniques for load balancing in parallel processing environments are inadequate, as they often fail to accurately detect skew due to reliance on comparisons between highest and lowest loads, accumulated statistics, or wall clock time, leading to delayed or missed detection of load imbalance issues.
Innovation Solution
A method for detecting skew in a parallel processing environment by calculating the difference between the highest loaded processor and an average load across remaining processors, with a skew threshold value and qualification time to determine if a load imbalance exception should be raised, using metrics like processing cycles or I/O counts, and measuring time in CPU cycles rather than wall clock time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If comparison is made between highest and lowest loaded processors to detect skew, then skew detection can be performed, but false positives occur when one processor has little or no work
Solution Approach 1:
The patent extracts the problematic comparison element (lowest loaded processor) from the skew detection process. Instead of comparing highest vs lowest loaded processors, the system only compares the highest loaded processor against the average load of all processors, eliminating the source of false positives while maintaining skew detection capability
Solution Approach 2:
The patent inverts the traditional skew detection approach by comparing the highest loaded processor against the average rather than the lowest. This inversion changes the reference point from an extreme value (lowest) to a central tendency measure (average), fundamentally altering how skew is measured and reducing false positives
2Quantity of substance
If accumulated statistics from the start of operation are used for skew detection, then historical data is captured, but detection is delayed or problems go undetected when imbalance occurs mid-operation
Solution Approach 1:
The patent implements periodic skew detection during operation execution, checking for load imbalance at regular intervals rather than only at the start. This periodic monitoring ensures that skew conditions developing mid-operation are detected promptly, reducing detection delay while maintaining reasonable overhead
Solution Approach 2:
The patent performs preliminary skew checks at structured points during operation execution, proactively detecting load imbalance before it significantly impacts performance. By checking at predetermined intervals rather than waiting for operation completion, the system anticipates and addresses skew conditions earlier
3Ease of manufacture
If wall clock time is used to determine when skew is present, then implementation is simple, but effectiveness is reduced on busy multi-user systems where operations may not run again during elapsed time
Solution Approach 1:
The patent changes the time measurement parameter from wall clock time to CPU time or operation-based intervals. This parameter change makes the detection mechanism adaptive to system load conditions, as CPU time continues to accumulate and provide meaningful measurement intervals even when wall clock time passes without operation execution
Data Source
AI summary
Techniques are presented for skew exception detection within a parallel processing environment. A potential exception is detected when a highest load of a processor within the parallel processing environment exceeds an average load within the parallel processing environment by a given threshold. The potential exception can become a legitimate exception when it exists for a qualified period or time or when it exists and no qualified period of time is noted. In an embodiment, checks for the potential exception occur and are wholly contained within a given configurable interval.