Precompiled ML Model Executables for Low Latency Inference

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current systems face challenges in facilitating high frequency processing due to limitations in low latency and increased processing demand, leading to lag and bottlenecking, especially in handling large-scale inputted data for real-time inference and transaction compliance.

Innovation Solution

A system that stores machine learning models as executable files on local memory, using a programming interpreter to create model executable files, allowing for parallel processing of streaming data without straining the processor, and enabling simultaneous execution for multiple data sets.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Speed

If machine learning models are processed using current systems, then processing capability is provided, but latency increases and processing speed decreases

Engineering Contradiction:

Improveprocessing speedVSAvoidlatency

Core Design Contradiction:

SpeedVSLoss of time

Solution Approach 1:

The patent pre-compiles machine learning model code into executable files before they are needed for processing. This preliminary compilation action stores the models in an optimized executable format that can be directly executed without interpretation overhead during runtime, thereby reducing latency and improving processing speed in high-frequency trading scenarios

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent replaces the traditional Python interpreter-based execution mechanism with a pre-compiled executable file execution mechanism. By substituting the interpretation process with direct executable file execution, the system eliminates the latency associated with code interpretation and achieves faster model processing speeds

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Productivity

If more processing power is used to handle large-scale data, then processing capability improves, but system complexity and resource demands increase

Engineering Contradiction:

Improveprocessing capabilityVSAvoidsystem complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent extracts the computationally intensive model compilation process from the runtime processing system. By separating the compilation phase (which creates executable files) from the execution phase (which runs pre-compiled models), the system reduces runtime complexity and resource demands while maintaining high processing capability through efficient executable file execution

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent changes the operational parameters of machine learning model execution by transitioning from an interpreted execution mode to a pre-compiled executable mode. This parameter change in the execution mechanism allows the system to handle large-scale data with reduced computational overhead and simplified system architecture during runtime

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS20230260025A1System and method for facilitating high frequency processing using stored models

Publication Date: 2023.08.17 BANK OF AMERICA CORP

AI summary

Systems, methods, and computer program products are provided for facilitating high frequency processing using stored models. A method for facilitating high frequency processing using stored models is provided. The method includes receiving a set of code relating to a machine learning model configured to process data. The method also includes generating a model executable file from the set of code relating to the machine learning model. The model executable file is configured to process inputted data using the machine learning model upon execution. The method still further includes storing the model executable file on an in-memory of a local device used to process inputted data.