Prompt Suitability Analysis for LLM Security
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models (LLMs) face challenges in generating accurate and reliable responses due to hallucinations, factual inaccuracies, and the inability to process unsuitable or malicious prompts, which can lead to diminished user experiences and increased costs for evaluation and training.
Innovation Solution
Implementing a prompt analysis system that evaluates the suitability of user prompts before processing them through LLMs. This system uses the LLM's training data to determine the likelihood of prompt types and generates a verification prompt to assess the prompt's suitability, thereby filtering out unsuitable or malicious prompts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If LLMs process all user prompts without prior analysis, then processing speed is maintained, but quality and reliability of outputs deteriorate due to hallucinations and inability to handle unsuitable prompts
Solution Approach 1:
The patent applies preliminary action by performing prompt suitability analysis before the main LLM processing. The system evaluates whether a prompt is suitable for processing by checking against training data characteristics and generating verification prompts, filtering out unsuitable prompts before they reach the main LLM processing pipeline, thereby improving reliability without significantly impacting overall processing speed
2Measurement precision
If specialized arbiter models are used to evaluate prompt suitability, then measurement precision of prompt quality improves, but device complexity increases
Solution Approach 1:
The patent applies self-service by enabling the LLM to evaluate its own prompts using its existing training data and capabilities. The system generates verification prompts based on the original prompt and uses the LLM's inherent knowledge to assess suitability, eliminating the need for separate specialized arbiter models and reducing system complexity while maintaining evaluation precision
3Object-affected harmful factors
If prompt analysis and verification are performed, then loss of information from filtering malicious prompts increases, but harmful factors affecting the system decrease
Solution Approach 1:
The patent applies feedback by implementing a verification mechanism where the LLM generates verification prompts based on the original prompt and compares the responses to assess suitability. This feedback loop allows the system to make informed decisions about prompt acceptance or rejection, reducing false positives and minimizing the loss of valid information while still filtering malicious inputs
Data Source
AI summary
Disclosed are apparatuses, systems, and techniques that evaluate suitability of prompts for language model (LM) processing for improved quality and security of LM outputs. The techniques include determining prompt verification score(s) that include a first subset of tokens and a second subset of tokens, and obtaining, using an LM, the individual prompt verification score characterizing a likelihood that the second subset of tokens occurs, in the prompt, together with the first subset of tokens. The techniques further include determining, using the prompt verification score(s), whether the prompt is to be provided to the LM.