Security-Deceptive Content Detection With LVM Prompt Verification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing large language models (LLMs) and large vision models (LVMs) face challenges in accurately processing complex visual and textual data, particularly in domain-specific tasks, and are prone to hallucinations, which are costly and resource-intensive to mitigate, limiting their effectiveness in high-stakes scenarios like scam detection and cybersecurity.
Innovation Solution
A computer-implemented method and system that utilizes state-of-the-art LVMs and LLMs, leveraging Retrieval-Augmented Generation (RAG) with black-box access, to extract features and classify risks by integrating domain-expert knowledge, applying risk rulesets and hallucination mitigation techniques, ensuring accurate and efficient processing of textual and visual data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If state-of-the-art LVMs are used for feature extraction, then processing speed is improved, but accuracy deteriorates in domain-specific tasks
Solution Approach 1:
The system segments the processing pipeline into two distinct stages: a first stage using LVMs for rapid initial feature extraction and classification, and a second stage using domain-specific trained models for accurate analysis of suspected malicious content. This segmentation allows each model to operate in its optimal performance zone, with LVMs providing speed for initial triage and specialized models ensuring accuracy for critical assessments.
Solution Approach 2:
The system introduces an intermediary mechanism in the form of a backend controller that manages the interaction between LVMs and domain-specific models. This intermediary coordinates the workflow by receiving outputs from the LVM stage, filtering for cases requiring specialized analysis, and routing to appropriate domain-specific models, thereby bridging the performance gap between general-purpose and specialized models.
2Measurement precision
If domain-specific trained models are used, then accuracy is improved, but resource consumption increases
Solution Approach 1:
The system applies partial action by using resource-intensive domain-specific models only for a subset of cases where the LVM stage identifies potential malicious content, rather than applying them to all inputs. This selective deployment ensures high accuracy where needed while minimizing overall resource consumption by avoiding unnecessary processing of benign content through the more demanding specialized models.
3Ease of manufacture
If LVMs are used for scam detection, then cost is reduced, but reliability deteriorates due to hallucinations
Solution Approach 1:
The system implements beforehand cushioning by using the LVM stage as a preliminary filter that identifies and isolates potentially problematic content before it reaches the more reliable domain-specific models. This protective mechanism prevents hallucinations in the specialized models from propagating by ensuring they only process content that has already been flagged as suspicious, thereby maintaining reliability while keeping costs manageable through the use of cheaper LVMs for initial processing.
Data Source
AI summary
Detection of a malicious application or website by a transaction processing application includes inputting a transactional request having a transactional data record and a screenshot; sending the input data record and input screenshot to a backend controller; requesting to a prompt selector, a string comprising a feature-extraction prompt; sending the input screenshot and the received prompt string to a Large Vision Model, LVM; receiving a string having risk classification features from said LVM; verifying the received string by a format parser; if the received string fails the verification, requesting by the backend controller, a string having a feature-extraction prompt which explicitly mentions format parsing compatibility, and repeating the preceding steps; sending the received string to a risk classification model for providing a risk classification; sending the risk classification to the backend controller; determining if the application or website is determined as malicious, and accepting or rejecting the transactional request accordingly.