Trace-Based Compilation for Faster Dynamic Model Execution

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing machine learning frameworks face inefficiencies in compilation time, lack of support for dynamic models, and debugging complexity, particularly in eager execution frameworks, while graph-based frameworks miss optimization opportunities due to static compilation.

Innovation Solution

A hybrid system that intercepts function calls from an eager execution framework, compiles operations in traces, and uses a trace manager to optimize compilation by selecting appropriate compilers and devices based on statistical analysis and caching, allowing for efficient compilation and execution of dynamic programs.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Speed

If graph-based framework compiles entire program in advance, then execution speed is improved, but compilation time increases and support for dynamic models deteriorates

Engineering Contradiction:

Improveexecution speedVSAvoidcompilation time

Core Design Contradiction:

SpeedVSLoss of time

Solution Approach 1:

The system segments the program compilation process by dividing the entire program into smaller trace instances that can be compiled independently. The trace manager collects operations into traces, selects relevant trace instances for compilation, and compiles them separately rather than compiling the entire program at once, thus reducing compilation time while maintaining execution speed benefits.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system performs preliminary actions by pre-compiling frequently executed trace instances. The trace manager identifies hot traces that are executed frequently and compiles them in advance, so that when these traces are encountered during execution, the compiled code is already available, improving execution speed without requiring full program compilation.

Inventive Principle:

Principle #10Preliminary action

2Productivity

If graph-based framework compiles entire program in advance, then execution efficiency is improved, but adaptability to dynamic models deteriorates

Engineering Contradiction:

Improveexecution efficiencyVSAvoidsupport for dynamic models

Core Design Contradiction:

ProductivityVSAdaptability or versatility

Solution Approach 1:

The system introduces dynamics by allowing the compilation process to adapt to runtime conditions. The trace manager monitors execution, identifies frequently executed traces dynamically, and decides which traces to compile based on runtime statistics. This dynamic approach enables the system to handle dynamic models effectively while maintaining execution efficiency through selective compilation of relevant trace instances.

Inventive Principle:

Principle #15Dynamics

3Adaptability or versatility

If eager execution framework executes operations sequentially, then support for dynamic models is improved, but execution speed deteriorates

Engineering Contradiction:

Improvesupport for dynamic modelsVSAvoidexecution speed

Core Design Contradiction:

Adaptability or versatilityVSSpeed

Solution Approach 1:

The trace manager acts as an intermediary between the eager execution framework and the compiler. It intercepts operations from the eager framework, groups them into traces, selects relevant trace instances, and passes them to the compiler for compilation. This intermediary layer enables the eager framework to maintain its dynamic execution capabilities while the compiler optimizes specific trace instances for faster execution.

Inventive Principle:

Principle #24Intermediary (Mediator)

4Speed

If compiler compiles operations in advance, then execution speed is improved, but device resource utilization deteriorates

Engineering Contradiction:

Improveexecution speedVSAvoiddevice resource utilization

Core Design Contradiction:

SpeedVSQuantity of substance

Solution Approach 1:

The system applies local quality by optimizing only the specific trace instances that are frequently executed and relevant to the current execution context, rather than compiling the entire program. The trace manager analyzes runtime statistics and selects trace instances for compilation based on their local importance and frequency, thus improving execution speed for critical paths while minimizing device resource utilization.

Inventive Principle:

Principle #3Local quality

Data Source

PatentEP4363966B1Compilation system and method

Publication Date: 2025.07.30 MICROSOFT TECHNOLOGY LICENSING LLC

Figure 1
Figure 2
Figure 3~4

AI summary

A dispatcher steps through a sequence of operations in a source program, including looping through the loops and/or branching at the branches, and maps each source operation to a respective function call each time encountered. An interceptor intercepts the function calls, converts each into a corresponding trace entry comprising a set of one or more trace operations, and inserts the trace entries one after another into the trace buffer to thereby form a linear trace of the source program. A trace manager transfers trace instances from the trace buffer to at least one compiler for compilation into corresponding compiled code, each trace instance comprising a current instance of the trace in the trace buffer or a portion thereof. The trace manager receives back the compiled code compiled from each trace instance, and sends it on to at least one destination device to be executed.