Remote Session Anomaly Detection via OCR and LLM
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Monitoring RDP/SSH sessions in real-time for anomaly detection poses challenges such as performance degradation, user privacy concerns, compliance with privacy laws, and accuracy issues like false positives and false negatives.
Innovation Solution
The AIDA system uses vaulted credentials for secure remote access and employs a computer vision algorithm with AI for real-time monitoring. It transcribes session activities into text using OCR and processes this text with a Large Language Model (LLM) to detect anomalies, minimizing direct video processing by AI.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If real-time monitoring of RDP/SSH sessions is implemented using computer vision algorithms, then anomaly detection capability is improved, but performance degradation occurs
Solution Approach 1:
The patent extracts only the essential information from video frames using optical character recognition (OCR) technology, rather than processing the complete video streams. By extracting text elements, commands, and interface components from screenshots, the system achieves anomaly detection while minimizing the computational load on the remote session, thus resolving the contradiction between monitoring capability and performance degradation.
2Reliability
If continuous screenshot capture and video processing is performed, then real-time anomaly detection is improved, but data privacy concerns and compliance issues worsen
Solution Approach 1:
The system extracts only textual information from visual frames using OCR, eliminating the need to process, store, or transmit complete video recordings. This extraction approach maintains real-time monitoring capability while significantly reducing data privacy risks, as only minimal text data is handled rather than full video streams.
Solution Approach 2:
The patent introduces an intermediary processing layer that analyzes screenshots through OCR and compares them against baseline behavior patterns. This intermediary approach allows anomaly detection without direct access to continuous video data, thereby addressing privacy concerns while maintaining security monitoring.
3Measurement precision
If computer vision algorithms process video frames for anomaly detection, then detection accuracy is improved, but false positives and false negatives increase
Solution Approach 1:
The system performs preliminary action by capturing baseline screenshots during normal session operation and storing them as reference data. During actual monitoring, new screenshots are compared against these pre-established baselines to detect deviations. This preliminary baseline capture enables more accurate anomaly detection by providing context-specific reference points, thereby reducing false positives while maintaining high detection accuracy.
Data Source
AI summary
A method for reducing the risk of unexpected or unintended actions during a Remote Desktop Protocol (RDP) or SSH (Secure Shell) session by automatically detecting anomalies using vaulted credentials. The method includes initiating a RDP/SSH session on a server and taking screenshots of the session. The screenshots are sent to a system that uses a computer vision algorithm to transcribe the screen information into a text-based transcription of the session. The text-based transcription of the session recording is then provided to a pre-trained LLM which is prompted to look for anomalies using in-context learning. The detected anomalies are presented to a system administrator or sent as alerts to the system administrator.