Distributed LLM Input Chunking for Lower-Power Multi-Device Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Implementing large generative AI models (LXMs) on a single computing device requires significant resources, leading to high costs and power consumption, especially for battery-powered devices, and existing methods do not efficiently distribute these models across multiple devices to balance execution time and reduce resource usage.
Innovation Solution
Distribute the LXM across a cluster of computing devices by dividing it into portions based on device characteristics and model structure, processing input chunks in parallel, and balancing execution time across devices, using input chunking and parallel processing to optimize resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If LXM is implemented on a single computing device, then model functionality is maintained, but resource consumption and power consumption increase significantly
Solution Approach 1:
The patent divides the LXM into multiple portions distributed across different computing devices. Each computing device holds and processes a specific portion of the model, enabling the system to maintain full model functionality while reducing the resource and power consumption on any single device. This segmentation allows battery-powered devices to participate in LXM processing without requiring excessive power.
2Use of energy by moving object
If LXM is distributed across multiple computing devices, then resource consumption per device decreases, but system complexity increases
Solution Approach 1:
The LXM is segmented into portions that are distributed across multiple computing devices. Each device manages only its assigned portion, which simplifies individual device complexity while collectively providing the full model capability. The system architecture handles the distribution and coordination complexity at a higher level.
Solution Approach 2:
The computing devices are designed to be universal participants in the distributed LXM system. Each device can execute any portion of the model regardless of its specific hardware configuration, and the system automatically assigns portions based on device characteristics. This multi-functionality reduces the need for device-specific optimization and simplifies the overall system design.
3Productivity
If input is processed in chunks across devices, then execution time is balanced and resource utilization is optimized, but processing steps and communication overhead increase
Solution Approach 1:
The input is divided into chunks that are processed in parallel across different computing devices. Each device processes its assigned chunk simultaneously, significantly reducing total execution time. The segmentation of input processing steps allows for balanced workload distribution and optimized resource utilization across the distributed system.
Solution Approach 2:
The system performs preliminary actions by pre-dividing the input into chunks and pre-assigning processing tasks to different devices based on their characteristics. This preparation enables efficient parallel processing and reduces the complexity of coordination during actual execution, as the task distribution is already optimized before processing begins.
Data Source
AI summary
Various embodiments include systems and methods for distributing a large generative AI model (LXM) across computing devices and implementing the LXM distributed across the computing devices. Embodiments may include identifying an input chunk size based on the characteristics, dividing an input into input chunks of the input chunk size. Embodiments may include processing input chunks by executing a portion of the LXM generating intermediary chunks, transmitting the intermediary chunks to another computing device configured to process the intermediary chunks by executing another portion of the LXM, and processing other input chunks by executing the portion generating other intermediary chunks in parallel with transmitting the intermediary chunks.