Ensemble Cluster Tuning for AI Accelerator Telemetry

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current deep learning frameworks face inference failures and performance issues due to inaccurate compute and memory requirement estimations, leading to runtime failures and suboptimal performance, especially when clusters are executed on accelerated backends without considering telemetry and temperature metrics.

Innovation Solution

An ensemble-based subgraph partitioning solution that utilizes telemetry, temperature, and compute metrics to automatically determine the best backend for executing AI model clusters, preventing inference failures and improving performance by dynamically selecting between accelerated and default backends based on real-time data.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Speed

If clusters are executed on accelerated backend using heuristic estimation, then execution speed is improved, but runtime failures occur due to inaccurate compute and memory requirement estimations

Engineering Contradiction:

Improveinference execution speedVSAvoidruntime execution reliability

Core Design Contradiction:

SpeedVSReliability

Solution Approach 1:

The system implements feedback by continuously monitoring telemetry data (compute metrics, memory usage, temperature) from previous cluster executions on the accelerated backend. This feedback loop allows the system to learn from past performance and adjust cluster allocation decisions dynamically, preventing runtime failures by identifying patterns that lead to execution errors and avoiding similar allocations in the future.

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The system performs preliminary analysis of telemetry data and temperature metrics before allocating clusters to the accelerated backend. By evaluating compute requirements and thermal conditions in advance, the system can make informed decisions about cluster execution, preventing runtime failures before they occur rather than reacting to them after execution begins.

Inventive Principle:

Principle #10Preliminary action

2Productivity

If more clusters are allocated to accelerated backend, then inference performance is improved, but hardware temperature increases reducing system longevity

Engineering Contradiction:

Improveinference performanceVSAvoidhardware temperature

Core Design Contradiction:

ProductivityVSTemperature

Solution Approach 1:

The system dynamically adjusts cluster allocation to the accelerated backend based on real-time temperature metrics and thermal conditions. When temperature thresholds are approached, the system automatically reduces or pauses cluster execution on the accelerated backend, allowing the hardware to cool down. This dynamic adaptation maintains optimal performance within safe thermal boundaries, extending hardware longevity while maximizing inference throughput during cooler periods.

Inventive Principle:

Principle #15Dynamics

3Reliability

If telemetry monitoring is implemented for cluster allocation decisions, then execution reliability is improved, but system complexity increases

Engineering Contradiction:

Improvecluster execution reliabilityVSAvoidsystem architecture complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The system implements a multi-functional telemetry collection mechanism that gathers compute metrics, memory usage data, and temperature information through a unified interface. This universal approach consolidates multiple monitoring functions into a single framework, reducing the complexity that would otherwise arise from separate monitoring systems for each metric type.

Inventive Principle:

Principle #6Universality (Multi-functionality)

4Productivity

If framework partitions graph to send supported subgraphs to accelerated backend, then execution efficiency is improved, but unsupported operators cause fallback failures

Engineering Contradiction:

Improvegraph execution efficiencyVSAvoidoperator compatibility reliability

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

Instead of the traditional approach of identifying unsupported operators and falling back to CPU execution, the system inverts the logic by proactively identifying and isolating operators that would benefit from accelerated backend execution. The system pre-configures operator-to-backend mappings based on telemetry data and capability profiles, ensuring that only compatible operators are routed to the accelerated backend, thereby eliminating fallback failures while maintaining high execution efficiency.

Inventive Principle:

Principle #13The other way round (Inversion)

Data Source

PatentUS11640326B2Ensemble based cluster tuning and framework fallback for AI accelerators using telemetry, compute, and temperature metrics

Publication Date: 2023.05.02 INTEL CORP

AI summary

Systems, apparatuses and methods may provide for technology that identifies telemetry data associated with an execution of a cluster of artificial intelligence (AI) operations on an accelerated backend system, wherein the telemetry data includes one or more of temperature classifier data, compute classifier data or failure data, and determines whether to send a current instance of the cluster of AI operations to the accelerated backend system or a default backend system based on the telemetry data.