AI Gateway Rule Engine for Secure LLM Routing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Enterprises face challenges in using large language models (LLMs) due to concerns over sensitive data exposure and inefficiencies in model routing, leading to hesitancy in adopting AI technologies, with existing solutions failing to address both data security and model routing issues.
Innovation Solution
A rule engine that performs data loss prevention (DLP) and AI model routing, allowing enterprises to define granular management rules for sensitive data handling and intelligent model selection, integrated with a gateway for easy implementation, which applies rules to inputs and outputs, and routes queries to appropriate AI models based on context and provider restrictions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If enterprises use LLMs to handle queries, then productivity and efficiency are improved, but sensitive data security is compromised
Solution Approach 1:
The patent introduces a gateway as an intermediary component between the enterprise application and the LLM service. This gateway enforces management rules that prevent sensitive data from being transmitted to external LLM providers while still allowing the LLM to process non-sensitive queries. The gateway acts as a protective mediator that enables productivity improvement without compromising data security.
Solution Approach 2:
The system performs preliminary evaluation of inputs against management rules before they are sent to the LLM. The gateway evaluates whether an input contains sensitive data or violates provider restrictions before allowing it to reach the LLM service. This preliminary action prevents sensitive data exposure while maintaining the ability to use LLMs for efficient query handling.
2Device complexity
If a single AI model is used for all queries, then device complexity is reduced, but productivity decreases due to inappropriate model handling
Solution Approach 1:
The patent implements dynamic model routing where the gateway automatically selects the most appropriate AI model based on the characteristics of each input query and enterprise management rules. This dynamic selection process optimizes query processing efficiency by routing different types of queries to different models without requiring the enterprise to manually manage multiple models or complex routing logic.
Solution Approach 2:
The gateway performs self-service model routing by automatically evaluating inputs against management rules and selecting appropriate models without requiring enterprise intervention. The system autonomously handles the complexity of model selection, allowing the enterprise to benefit from optimized query processing while maintaining simple system architecture.
3Productivity
If enterprises adopt AI services without restrictions, then productivity is improved, but compliance with provider terms and data security policies deteriorates
Solution Approach 1:
The gateway implements feedback mechanisms that continuously evaluate inputs and outputs against management rules and provider restrictions. When violations are detected, the system provides feedback by blocking the input, modifying it, or routing it to an appropriate model. This feedback loop ensures compliance with provider terms and data security policies while maintaining high AI service utilization for compliant queries.
Solution Approach 2:
The system applies preliminary anti-action by proactively preventing compliance violations before they occur. The gateway evaluates inputs against management rules and provider restrictions before they reach the LLM service, blocking or modifying potentially non-compliant inputs in advance. This preliminary prevention ensures compliance assurance while allowing unrestricted AI service adoption for compliant queries.
Data Source
AI summary
The invention provides a rules engine that evaluates, modifies, and dynamically routes inputs to AI models based on provider restrictions and management rules. Provider restrictions are stored. Based on management rules and the detected provider restrictions, the input can be modified with contextual placeholders and prompts can be injected for input into the destination model. Reversal of the modifications can occur to the outputs. The system can notify the user, an administrator, and a supervisor regarding the security evaluation and remedial actions. The evaluations can be logged for auditing purposes.