Tree Data Structure for High-Dimensional Security Log Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High-dimensional data logs in information security and information technology pose challenges in identifying unique or anomalous events due to the large memory requirements and tradeoff between memory allocation and lookback capacity, making it inefficient to search for unusual events.
Innovation Solution
A tree data structure is used to decouple indexing of variables, filtering out high-frequency relationships by replacing the indexed table with a tree structure where each variable is associated with a different level, reducing memory footprint and increasing lookback capacity while maintaining significant dependence between source host and user variables.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If an indexed table data structure is used to store high-dimensional data logs, then memory allocation is straightforward, but the memory footprint becomes excessively large and lookback capacity is limited
Solution Approach 1:
The patent segments the flat indexed table into a hierarchical tree structure where data is organized by variable types (e.g., source host, user, target host) at different levels. This segmentation allows the system to store only relevant portions of high-dimensional data relationships, reducing overall memory footprint while maintaining the ability to traverse and analyze data over extended lookback periods.
Solution Approach 2:
The patent transforms the two-dimensional indexed table into a multi-dimensional tree structure with additional hierarchical levels representing different variable types and relationships. This dimensional transformation enables more efficient storage by exploiting the hierarchical nature of security event data, allowing the system to reduce memory requirements while preserving lookback capacity through the added structural dimension.
2Loss of information
If all variables are indexed in the data structure, then complete data coverage is achieved, but memory requirements increase dimensionally with each variable
Solution Approach 1:
The patent extracts and separates different variable types into distinct hierarchical levels within the tree structure. By taking out commonly occurring variables (such as source host and user) and placing them at higher levels of the hierarchy, the system avoids redundant storage of these variables across multiple data entries, thereby reducing memory allocation while maintaining complete data coverage through the hierarchical organization.
Solution Approach 2:
The patent applies local quality by treating different variable types differently within the hierarchy. Frequently occurring variables are stored at higher levels with greater sharing, while less frequent variables are stored at lower levels. This localized optimization ensures complete data coverage is maintained where needed while reducing memory allocation through selective optimization at different hierarchical levels.
3Quantity of substance
If a reduced data set is used to decrease memory footprint, then storage requirements are reduced, but the ability to detect unusual events may be compromised
Solution Approach 1:
The patent performs preliminary organization of data into a hierarchical tree structure that pre-computes and stores aggregated information at each level. This preliminary action enables the system to maintain a reduced data footprint while preserving anomaly detection capability, as the hierarchical structure allows for efficient querying and analysis of patterns without requiring storage of all raw individual events.
Solution Approach 2:
The hierarchical tree structure acts as an intermediary between the raw high-dimensional data and the anomaly detection process. By introducing this intermediate hierarchical representation, the system can reduce storage requirements through aggregation and summarization at different levels while maintaining the ability to detect unusual events by traversing and analyzing the hierarchical relationships.
Data Source
AI summary
A value is assigned to a rate threshold for adding child nodes to a distinct parent node in a tree data structure. A first datum comprising a first variable assigned a first value and a second variable assigned a first value is added to the tree at a first timestamp, by adding to the first level in the tree a first parent node representing the first variable assigned the first value and adding to the second level in the tree a first child node representing the second variable assigned the first value and connected by a first directed edge from the first parent node. A second datum comprising the first variable assigned the first value and the second variable assigned a second value is received at a second timestamp. The method blocks adding to the second level in the tree a second child node representing the second variable assigned the second value and connected by a second directed edge from the first parent node when a rate based on the first timestamp and the second timestamp exceeds the rate threshold.