Mixed Precision Compiler for Computational Graphs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for executing machine learning and artificial intelligence programs on Von Neumann computers require high-precision data types, leading to increased computational load, energy consumption, and memory requirements, which limit performance and scalability, especially on power-limited hardware platforms, and manual intervention is often tedious and error-prone for optimizing mixed-precision techniques.
Innovation Solution
A graph-level automatic mixed-precision (graphAMP) technique that automatically adjusts floating-point data types across computational graphs, categorizing operations into three sets—deny, allow, and infer sets—to maintain high precision for sensitive operations while downcasting stable operations to lower precision, optimizing performance and energy efficiency without compromising accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If high-precision data types are used, then model accuracy is maintained, but computational load and energy consumption increase
Solution Approach 1:
The patent applies different precision levels to different operations within the computational graph based on their numerical stability characteristics. Numerically stable operations use lower precision (e.g., bfloat16) to reduce energy consumption, while numerically sensitive operations maintain higher precision (e.g., float32) to preserve model accuracy. This local differentiation resolves the contradiction by optimizing energy usage without sacrificing overall accuracy.
Solution Approach 2:
The patent dynamically changes the precision parameter of data types based on the operational context. By analyzing the computational graph and identifying which operations can tolerate lower precision, the system adjusts data type parameters (precision levels) to minimize energy consumption while maintaining acceptable accuracy thresholds.
2Measurement precision
If high-precision data types are used, then model accuracy is maintained, but computational speed decreases
Solution Approach 1:
The patent implements local quality by assigning different precision levels to specific operations based on their numerical stability. Numerically stable operations execute at lower precision with higher speed, while sensitive operations maintain higher precision. This resolves the speed-accuracy contradiction by allowing parallel execution of low-precision fast operations and high-precision accurate operations.
3Measurement precision
If manual intervention is used for mixed-precision optimization, then precision control is possible, but the process becomes tedious and error-prone
Solution Approach 1:
The patent implements self-service by automatically analyzing the computational graph, identifying numerically stable operations, and applying appropriate precision levels without user intervention. The system autonomously determines which operations can use lower precision while maintaining accuracy, eliminating the tedious and error-prone manual optimization process while preserving precision control through automated analysis.
Solution Approach 2:
The patent automatically changes precision parameters based on computational graph analysis. By programmatically identifying operations suitable for mixed precision and applying precision transformations, the system replaces manual parameter adjustment with automated parameter change, improving ease of operation while maintaining precision control.
4Use of energy by moving object
If lower precision is used, then energy efficiency improves, but model accuracy may be compromised
Solution Approach 1:
The patent applies local quality by differentiating between numerically stable and sensitive operations. Stable operations use lower precision for improved energy efficiency, while sensitive operations maintain higher precision to protect model accuracy. This spatial differentiation of precision levels resolves the energy-efficiency-accuracy contradiction.
Solution Approach 2:
The patent applies partial action by selectively reducing precision only for operations that can tolerate it (numerically stable operations), rather than applying uniform low precision across the entire computational graph. This partial application of low precision achieves energy savings without compromising overall model accuracy.
Data Source
AI summary
The disclosed technology relates to automatically optimizing the precision of data types in a computational graph, such as those used in machine learning and artificial intelligence applications. A representation of the computational graph is obtained. Nodes of the computational graph are assigned to one of three sets: a deny set, an allow set, or an infer set, based on a predefined policy. For nodes in the allow set, the method changes at least one of the input data precision, output data precision, or internal computation precision to a lower precision. For nodes in the infer set, the method propagates a data precision requirement from downstream nodes to upstream nodes. The method generates and stores computer instructions for executing the computational graph with the optimized precisions on one or more processors. This approach enhances performance and energy efficiency while maintaining model accuracy for the computational graph.