Parallel Iterative Solver Communication Overhead Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The parallelization of iterative methods for solving systems of linear equations using multiple processing units is bottlenecked by the time taken for inter-process communication, particularly in calculations involving large numbers of processes.
Innovation Solution
The proposed solution involves determining the timing of end processing for iterative method updates based on the number of solution updates and residual norm calculations, optimizing communication by reducing the frequency of allreduce operations and focusing residual norm calculations only when necessary, thereby hiding communication time within computation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If multiple processing units operate in parallel for iterative method calculations, then calculation speed is improved, but inter-process communication time increases and becomes a bottleneck
Solution Approach 1:
The patent implements periodic determination processing where the residual norm is calculated at specific intervals (every NI iterations) rather than at every iteration step. This periodic action reduces the frequency of synchronization operations while still ensuring convergence accuracy, thereby reducing the cumulative communication time bottleneck in parallel iterative calculations
Solution Approach 2:
The patent performs preliminary determination of the timing for residual norm calculations based on the number of solution updates. By pre-planning when communication operations should occur based on iteration counts, the system optimizes the schedule of synchronization events, allowing computation to proceed efficiently between predetermined communication points
2Measurement precision
If residual norm calculation is performed frequently to ensure convergence accuracy, then solution accuracy is improved, but calculation time increases
Solution Approach 1:
The patent calculates the residual norm periodically every NI iterations rather than at every iteration, reducing the total number of expensive norm calculations while still monitoring convergence. This periodic measurement approach maintains solution accuracy by checking convergence at regular intervals without the time penalty of continuous monitoring
Solution Approach 2:
The patent performs residual norm calculations only when necessary (at predetermined intervals and when convergence conditions need checking) rather than at every possible opportunity. This partial action approach provides sufficient convergence verification without the excessive computational overhead of continuous calculation
3Reliability
If allreduce operations are performed frequently for synchronization, then convergence determination is improved, but communication overhead increases
Solution Approach 1:
The patent performs allreduce operations periodically at predetermined intervals (every NI iterations) rather than at every iteration step. This periodic synchronization maintains convergence determination reliability by checking results at regular intervals while dramatically reducing the frequency of costly collective communication operations across parallel processing units
Solution Approach 2:
The patent preliminarily determines the timing and frequency of allreduce operations based on the iteration count and convergence criteria. By pre-establishing when synchronization should occur, the system minimizes unnecessary communication overhead while ensuring convergence is properly determined at appropriate checkpoints
Data Source
AI summary
A non-transitory computer-readable recording medium stores a calculation program for causing a computer to execute a process including: executing calculation of an iterative method for iterating update of a solution by using a plurality of processing circuits operating in parallel in one or each of a plurality of loop processing; executing the calculation of the iterative method by using the plurality of processing circuits in predetermined loop processing after the one or plurality of loop processing; and determining a timing of determination processing of determining update end in the calculation of the iterative method of the predetermined loop processing based on a number of times the solution is updated in the calculation of the iterative method of the one or each of the plurality of loop processing, wherein the determination processing includes processing of determining the update end based on a result of communication between the plurality of processing circuits.