ML Model Precision Partitioning for Efficient Memory Offloading
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional machine learning (ML) models utilize high-precision numerical formats like single-precision floating-point format (FP32), leading to significant computational overhead and memory usage, which compromises performance and introduces errors.
Innovation Solution
A method and system that partitions ML model states into optimized subgraphs, determines optimal bit-length for offloading to target precision, and performs memory management to enhance computational efficiency and performance by reducing precision without compromising accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If high-precision numerical formats (FP32) are used in ML models, then computational accuracy is maintained, but computational overhead and memory usage increase significantly
Solution Approach 1:
The patent segments the computational process into multiple stages with different precision requirements. Model states are partitioned into subgraphs and processed at different precision levels (FP32, FP16, INT8), allowing computationally intensive operations to be performed at lower precision while maintaining accuracy where needed. This segmentation resolves the contradiction by applying high precision only where necessary and lower precision where acceptable.
Solution Approach 2:
The patent dynamically changes the precision parameter of numerical data during computation. By converting model states between different precision formats (FP32 ↔ FP16 ↔ INT8) based on computational requirements and error thresholds, the system achieves high computational efficiency at lower precision while maintaining accuracy through selective precision conversion and error compensation techniques.
2Measurement precision
If high-precision numerical formats (FP32) are used in ML models, then computational accuracy is maintained, but memory usage increases significantly
Solution Approach 1:
The patent segments model states into different precision groups (high-precision and low-precision states) that are stored and processed separately. This allows the system to maintain FP32 accuracy only for critical model parameters while using FP16 or INT8 for others, significantly reducing overall memory usage while preserving computational accuracy where required.
Solution Approach 2:
The patent dynamically changes the precision parameter of stored model states based on their importance and error sensitivity. By converting between precision formats and using precision conversion techniques, the system reduces memory footprint while maintaining the ability to restore and maintain accuracy when needed during computation.
3Productivity
If model states are partitioned into subgraphs and offloaded to lower precision, then computational efficiency improves, but precision loss may occur
Solution Approach 1:
The patent applies different precision qualities to different parts of the model based on their error sensitivity. Critical model states that require high accuracy remain in FP32, while less sensitive states are offloaded to FP16 or INT8. This local differentiation of precision quality allows computational efficiency improvements without significant overall precision loss.
Solution Approach 2:
The patent implements feedback mechanisms through error threshold monitoring and adaptive precision adjustment. By continuously monitoring computation errors and adjusting precision levels dynamically, the system ensures that precision is maintained above acceptable thresholds while maximizing computational efficiency through lower precision where possible.
4Quantity of substance
If memory management techniques are applied to offloaded data, then memory usage is optimized, but system complexity increases
Solution Approach 1:
The patent segments memory management into specialized handlers for different precision types (FP32, FP16, INT8). Each precision type has dedicated memory allocation and management routines, which simplifies the overall management complexity by treating each precision category independently rather than managing all data uniformly.
Data Source
AI summary
Traditional machine learning operations utilize high-precision numerical formats that lead to computational overhead and memory usage. Hence, it is imperative that precision of these computations needs to be reduced without compromising accuracy of machine learning (ML) models. Present disclosure provides systems for optimizing computational efficiency and performance of machine learning (ML) models. ML model consisting of model states in an original precision is fed as input, which are analyzed to determine (i) how to best divide model states into smaller chunks for efficient parallel processing, and (ii) an optimal bit-length for offloading of model states to a target precision and obtain offloaded data. The offloaded data is transferred to computing node which performs broadcast operations to obtain result(s). The results are integrated into original precision based on a mapping of subset of partitioned model states to the offloaded data to obtain a validated integrated result.