On-Demand ML Inference Container API

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing machine learning serving infrastructures require significant overhead due to the use of continuous integration/deployment tools for spinning up and managing scoring containers, making it unsuitable for rapid testing and prototyping needs.

Innovation Solution

An application programming interface (API) is implemented to allow for the on-demand spinning up and spinning down of scoring containers, eliminating the need for continuous integration/deployment tools and enabling more agile development and testing processes.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If continuous integration/deployment tools are used to manage scoring containers, then deployment reliability is improved, but system complexity and overhead increase

Engineering Contradiction:

Improvedeployment reliabilityVSAvoidsystem complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent extracts the container management functionality from complex CI/CD toolchains and implements it directly within the machine learning serving infrastructure. The scoring container manager handles container lifecycle operations (creation, deployment, monitoring, termination) natively, eliminating the need for external CI/CD orchestration and reducing overall system complexity while maintaining deployment reliability.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The scoring container manager implements self-service capabilities by automatically managing the lifecycle of scoring containers without requiring manual intervention or complex external tooling. The system autonomously handles container creation, model deployment, performance monitoring, and termination based on demand signals, simplifying the deployment process while ensuring reliable operations.

Inventive Principle:

Principle #25Self-service

2Productivity

If scoring containers are spun up on demand, then development agility is improved, but resource management complexity increases

Engineering Contradiction:

Improvedevelopment agilityVSAvoidresource management complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent implements dynamic container provisioning where scoring containers are created, scaled, and terminated based on real-time demand signals from the machine learning serving infrastructure. The scoring container manager dynamically adjusts resource allocation and container lifecycle management, enabling rapid prototyping and model validation while automatically managing resource complexity through demand-driven operations.

Inventive Principle:

Principle #15Dynamics

Data Source

PatentUS20250068454A1Application programming interface for spinning up machine learning inferencing server on demand

Publication Date: 2025.02.27 SALESFORCE INC

AI summary

A method by one or more electronic devices for creating an inference container on demand. The method includes receiving, over a network, a request to create the inferencing container, wherein the inferencing container is configured to provide inferencing functionality, creating the inferencing container responsive to receiving the request to create the inferencing container, and providing, over the network, a response to the request to create the inferencing container, wherein the response includes a uniform resource locator (URL) to use to submit inferencing requests to the inferencing container, wherein the URL includes a unique identifier (ID) of the inferencing container.