Entity Maliciousness Analysis with Embedding-Based Log Sampling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional cybersecurity techniques struggle to detect novel malicious activities due to reliance on historical data and known attack patterns, and large language models (LLMs) face token limits when processing large datasets for malicious behavior detection.
Innovation Solution
Utilize AI-based entity maliciousness analysis through embedding and sampling techniques to select a representative sample of logs, identifying potentially and statistically anomalous data points, and trigger an AI model with a prompt that includes these samples and descriptions to determine malicious behavior without exceeding token limits.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all logs are included in the AI prompt for analysis, then the detection accuracy is improved, but the token limit of the AI model is exceeded
Solution Approach 1:
The patent extracts only the most relevant and representative logs from the complete log dataset. An embedding model generates vector representations of all logs, and the system selects logs with the largest distances from their cluster centroids (most anomalous/representative samples). This extraction approach reduces the quantity of logs included in the AI prompt while maintaining detection accuracy by focusing on the most informative samples.
Solution Approach 2:
The patent segments the complete log dataset into multiple clusters based on embedding similarity. Each cluster represents a category of similar log patterns. The system then selects representative samples from each cluster rather than including all logs. This segmentation allows the AI model to analyze a manageable subset of logs that collectively represent the entire dataset, resolving the contradiction between comprehensive analysis and token limits.
2Reliability
If conventional techniques are used for malicious activity detection, then the system operates based on existing knowledge, but novel attacks cannot be identified
Solution Approach 1:
The patent replaces conventional heuristic and rule-based detection mechanisms with an AI model that processes embedding representations of logs. The AI model can identify patterns and anomalies that deviate from known attack patterns, enabling detection of novel attacks. The system uses embedding models to transform logs into vector representations, allowing the AI to reason about log content semantically rather than through fixed rules, thus improving adaptability to new threats while maintaining reliability through systematic analysis.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Techniques are described herein that are capable of performing AI-based entity maliciousness analysis using embedding and sampling. A representative sample of data associated with an entity is selected by comparing embeddings that represent the data. A potentially anomalous data point is identified in at least a portion of the data based on a proximity of a node, which corresponds to the potentially anomalous data point, in a tree to a root node of the tree. A statistically anomalous data point is identified in representative sample data points, which define the representative sample, as a result of the statistically anomalous data point indicating an unexpected occurrence of an event. An AI model is triggered to determine whether the entity exhibits malicious behavior by providing an AI prompt, including the representative sample and a description of the potentially anomalous data point and the statistically anomalous data point, to the AI model.