Sequence Mining Data Reduction for Explainable Threat Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The vast quantity of database transaction data hampers meaningful analysis for malicious activity detection due to data size and interpretability issues, leading to potential inaccuracies and mispredictions.
Innovation Solution
Generate a reduced dataset by creating a sequence presence vector and using a sequence relations table to identify related sequences, then apply this dataset to a neural network model for malicious activity detection, maintaining explainability and reducing data size.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the vast quantity of database transaction data is used for malicious activity detection, then the detection coverage is improved, but the analysis efficiency and response time deteriorate due to data size
Solution Approach 1:
The patent extracts only the essential sequential patterns from the vast transaction data by comparing actual action sequences against a library of known malicious sequences. Instead of analyzing all raw data, the system extracts and matches only the relevant sequence patterns, thereby maintaining detection coverage while dramatically improving analysis efficiency
Solution Approach 2:
The patent segments the large dataset into discrete action sequences that can be independently compared against known malicious patterns. By breaking down the continuous stream of transactions into segmented sequences of actions, the system enables efficient pattern matching without processing the entire dataset as a monolithic block
2Reliability
If the vast quantity of database transaction data is used for malicious activity detection, then the detection coverage is improved, but the response time deteriorates due to data size
Solution Approach 1:
The system extracts only the critical sequence patterns needed for detection by comparing actual user actions against a predefined library of malicious sequences. This extraction approach maintains comprehensive detection coverage while reducing the time required for analysis by focusing only on relevant patterns rather than processing all raw transaction data
3Quantity of substance
If naive attempts to reduce data size are made, then the data size is reduced, but data interpretability deteriorates leading to inaccuracies and mispredictions
Solution Approach 1:
The patent creates a simplified copy of the data in the form of sequence presence vectors that represent the actual transaction data in terms of known malicious patterns. This copying approach reduces data size while preserving interpretability because each vector element directly corresponds to the presence or absence of a specific known malicious sequence, maintaining a clear link to the original data meaning
Solution Approach 2:
The patent transforms the raw transaction data into a different parameter space by converting sequences of actions into sequence presence vectors. This parameter transformation changes the data representation from detailed action logs to compact binary vectors indicating pattern presence, thereby reducing size while maintaining interpretability through the direct mapping to known malicious patterns
4Quantity of substance
If direct latent space transformations are used for data reduction, then the data size is reduced, but accuracy deteriorates due to loss of explainability
Solution Approach 1:
Instead of transforming data into an opaque latent space, the patent creates an explainable copy in the form of sequence presence vectors that directly represent the presence of known malicious patterns. This copying method reduces data size while preserving accuracy because the reduced representation maintains a direct, interpretable link to the actual malicious sequences detected in the original data
Data Source
AI summary
A method and related system operations include obtaining a time-ordered set of action types and generating a first dataset by determining, for each respective stored sequence of a plurality of stored sequences, a respective dataset element indicating that the respective stored sequence is present in the time-ordered set of action types. The method may also include generating a reduced dataset based on the first dataset by detecting that the first dataset indicates that a first sequence and a second sequence are present in the time-ordered set of action types, determining a reduced dataset element based on the detection of a presence of the first sequence and the second sequence in the time-ordered set of action types and a score between the first sequence and the second sequence indicated by a table, and detecting malicious activity using a decision model based on the reduced dataset.