Multi-Device LLM Inference With Intra-Layer Parallel Synchronization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing size of large language models leads to significant computational costs for training and inference, with GPUs being inefficient for inference due to different operational characteristics and the need for optimal solutions tailored to inference operations.
Innovation Solution
A multi-device-based inference performance system that separates a large language model into matrix columns using intra-layer parallelism, enabling simultaneous matrix multiplication and All-Gather operations across devices, reducing communication overhead and latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If multiple accelerators are used for computation, then computational power is improved, but communication overhead and latency increase
Solution Approach 1:
The system segments the large language model into multiple partitions, with each accelerator device processing a specific partition. This allows parallel computation across multiple devices while managing communication overhead through structured data distribution and collection patterns.
Solution Approach 2:
The system performs preliminary data preparation and organization before computation, including pre-processing input data and organizing model parameters across devices. This reduces the communication overhead during the actual inference process by having data ready in the correct format and location.
2Measurement precision
If model size increases, then accuracy is improved, but computational cost increases
Solution Approach 1:
The system divides the large language model into multiple smaller partitions that can be distributed across multiple accelerator devices. Each device processes a portion of the model, enabling the system to handle large model sizes while distributing the computational cost across multiple devices rather than concentrating it on a single high-power device.
Solution Approach 2:
The system transitions from a single-device computational approach to a multi-device distributed approach, adding the dimension of spatial distribution. This allows the system to maintain high model accuracy while reducing the computational burden on any single device through parallel processing across multiple accelerators.
3Measurement precision
If data synchronization is performed frequently, then computation accuracy is improved, but communication overhead increases
Solution Approach 1:
The system performs preliminary organization of data and computation in a way that minimizes the frequency and volume of synchronization operations. By pre-processing data and structuring computations to maximize local processing, the system reduces the need for frequent inter-device communication while maintaining computation accuracy.
Solution Approach 2:
The system introduces an intermediary data structure and communication protocol that efficiently manages data exchange between devices. This intermediary layer optimizes the synchronization process by batching operations, compressing data where possible, and using efficient communication patterns that reduce the overall communication overhead.
Data Source
Figure 1(1)~1(5)
Figure 2
Figure 3
AI summary
Provided is a method and system for performing multi-device-based inference for a large language model. A multi-device-based inference performance system may include a plurality of devices configured to map to partitions that separate a large language model (LLM) according to an intra-layer parallelism method. Here, each of the plurality of devices may be implemented to synchronize data by sharing a sub-result of matrix multiplication on the data with another device of the plurality of devices while the matrix multiplication is being performed.