Large Model Fringe-Layer Compression for Network Bandwidth and Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning models are becoming increasingly resource-intensive, making deployment on a singular device unfeasible due to resource constraints, and distributed execution is affected by the performance of the connecting network.
Innovation Solution
Distribute pipelined layers of a machine learning model across multiple devices, identify fringe layers, and generate compressed fringe layers by reducing parameters, optimizing communication based on network performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If machine learning models are deployed on a single device, then model execution is simplified, but resource constraints make deployment unfeasible for increasingly resource-intensive models
Solution Approach 1:
The patent divides the machine learning model into multiple layers and distributes them across multiple devices in a pipeline architecture. Each device executes a subset of layers, transforming a single-device deployment problem into a multi-device collaborative system that accommodates resource-intensive models while maintaining manageable complexity at each node.
2Quantity of substance
If models are split for distributed execution across multiple devices, then resource constraints are addressed, but performance becomes dependent on network connection performance
Solution Approach 1:
The patent dynamically adjusts the granularity of layer distribution and compression parameters based on network conditions. By changing parameters such as layer split points, compression ratios, and buffer sizes, the system optimizes the balance between distributed execution benefits and network dependency drawbacks, maintaining reliable performance across varying network conditions.
3Loss of time
If fringe layers are compressed by reducing parameters, then bandwidth and latency are reduced, but model accuracy may be affected
Solution Approach 1:
The patent applies compression selectively to fringe layers (those at the boundaries of the pipeline) rather than uniformly across all layers. This partial action targets the layers that contribute most to communication overhead while preserving the accuracy-critical internal layers, achieving bandwidth and latency reduction without sacrificing model accuracy.
Data Source
AI summary
In one embodiment, a method herein comprises: distributing a plurality of pipelined layers of a machine learning model among a plurality of individual devices connected via a computer network; determining a subset of the plurality of pipelined layers that are fringe layers that interconnect between the plurality of individual devices via the computer network; generating compressed fringe layers by reducing parameters of the fringe layers; and executing the machine learning model with an input passed through the plurality of pipelined layers to produce an output, wherein communication between the plurality of individual devices is based on the compressed fringe layers.