AI Query Orchestration Using Metadata-Based Resource Routing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

The widespread use of AI resources as a 'one-stop shop' for user queries leads to increased costs and inefficiencies, as many queries can be optimized for non-AI computing resources, and AI resource availability often fluctuates, leading to unnecessary scaling and latency issues.

Innovation Solution

Implementing an AI taster that pre-processes user queries to determine suitable computing resources based on metadata, tagging queries for non-AI or AI processing, and using a load balancer to orchestrate deployment to the appropriate resources.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If AI resources are used for all user queries as a one-stop shop, then query processing capability is improved, but cost increases and efficiency decreases

Engineering Contradiction:

Improvequery processing capabilityVSAvoidcost

Core Design Contradiction:

Adaptability or versatilityVSLoss of energy

Solution Approach 1:

The system segments user queries into different categories using an AI taster that analyzes query characteristics and routes them to appropriate computing resources. Simple queries are handled by non-AI resources while complex queries requiring understanding and reasoning are routed to AI resources, eliminating the need to use AI resources for all queries.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

An AI taster acts as an intermediary component between users and computing resources. It pre-processes queries, determines the appropriate resource type needed, and routes queries accordingly. This intermediary enables efficient resource allocation without requiring all queries to go through expensive AI resources.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Reliability

If AI resources are scaled to meet increased demand, then availability is improved, but cost increases and latency issues occur

Engineering Contradiction:

ImproveavailabilityVSAvoidlatency

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The AI taster performs preliminary analysis of queries before they reach the main processing system. By pre-processing queries and determining routing decisions in advance, the system avoids latency that would occur if routing decisions were made later in the processing pipeline.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

Different parts of the system handle different types of queries with locally optimized processing. Simple queries are handled quickly by non-AI resources while complex queries receive AI processing, ensuring each query type gets the appropriate level of service without unnecessary delays.

Inventive Principle:

Principle #3Local quality

Data Source

PatentUS20260044381A1Orchestrating queries based on computing resource type

Publication Date: 2026.02.12 CISCO TECHNOLOGY INC

AI summary

This disclosure describes techniques for load balancing user queries for artificial intelligence (AI) processing. A user query may be received that is initially destined to be processed by an AI computing resource. The user query may be pre-processed to identify metadata associated with the user query (e.g., attributes, features, characteristics, etc. associated with a user prompt and/or input file of the user query). The metadata may be used to determine processing requirements associated with the user query. The processing requirements may be used to determine whether such processing is to be performed by a non-AI computing resource instead of an AI computing resource. The user query may be load-balanced accordingly, and subsequent output provided to a user in response to the user query.