ML Model Partitioning Across Process Domains to Cut Memory Swaps
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large machine learning models often exceed the memory capacity of individual process domains, leading to memory swap operations that degrade performance and introduce latencies due to memory thrashing.
Innovation Solution
The machine learning model is partitioned into subgraphs, each with defined inputs, outputs, and relationships, allowing the model to be instantiated across multiple process domains based on available memory, thereby minimizing memory swap overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a large machine learning model is deployed to a single process domain, then the model can be executed as a unified entity, but the model exceeds the memory capacity of the process domain, leading to memory swap operations and performance degradation
Solution Approach 1:
The machine learning model is divided into multiple subgraphs, each representing a portion of the model. These subgraphs are then distributed across multiple process domains, allowing each process domain to handle a manageable portion of the model that fits within its memory capacity, thereby eliminating memory swap operations while maintaining overall model execution integrity
Solution Approach 2:
The patent transitions from a single-process-domain execution model to a multi-process-domain execution model, adding the dimension of spatial distribution across multiple processes. This allows the model to be executed in parallel across multiple domains with coordinated memory management, reducing inference latency while maintaining execution integrity
2Device complexity
If a large machine learning model is deployed to a single process domain, then the model structure remains simple to manage, but memory thrashing occurs due to exceeding memory capacity, degrading performance
Solution Approach 1:
The model is segmented into subgraphs that can be independently managed and deployed across multiple process domains. Each subgraph is a self-contained unit with defined inputs and outputs, making it manageable despite the increased overall system complexity. This segmentation enables efficient parallel execution and memory management across domains
Solution Approach 2:
The patent creates a universal framework for deploying machine learning models across multiple process domains with different memory capacities. The graph-based representation and coordinated instantiation mechanism provide a multi-functional solution that can adapt to various deployment scenarios, from single-domain to multi-domain configurations, optimizing computational efficiency for each case
3Quantity of substance
If memory swap operations are used to handle large models, then the model can be executed beyond available memory, but memory swap overhead increases and performance degrades
Solution Approach 1:
By segmenting the model into subgraphs and distributing them across multiple process domains with sufficient combined memory capacity, the system eliminates the need for memory swap operations. Each process domain maintains its subgraphs in memory without swapping, thereby reducing memory swap overhead and energy loss while supporting large model sizes
Solution Approach 2:
The patent introduces an intermediary coordination mechanism that manages the instantiation and execution of subgraphs across multiple process domains. This intermediary layer coordinates memory usage and data flow between domains, enabling efficient handling of large models without relying on memory swap operations, thus reducing the associated energy overhead
Data Source
AI summary
Certain aspects provide techniques and apparatuses for efficient operation of a machine learning model based on partitioning the machine learning model. An example method generally includes receiving a graph for a machine learning model. The graph for the machine learning model generally includes a plurality of subgraphs representing different portions of the machine learning model. The machine learning model is instantiated across a plurality of process domains associated with a same application based on the plurality of subgraphs in the graph for the machine learning model. An inference is generated based on executing the machine learning model across the plurality of process domains, and one or more actions are taken based on the generated inference.