Session Table Partitioning for Network Security State Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing security information/event management (SIEM) systems face challenges in accurately tracking and maintaining changing state data, leading to false negatives and false positives due to transient IP addresses and frequent device state changes, which complicates the identification and investigation of network attacks.
Innovation Solution
The implementation of session tables that store and manage session information with key, value, and timestamp fields, allowing for efficient querying and correlation of security events with session data, using filters to reduce unnecessary lookups and employing rules to implicitly determine session boundaries, and periodic partitioning and rollup of session data to maintain real-time access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If session data is continuously updated and queried in real-time to track changing network state, then the ability to identify attacks improves, but the computational resources and system complexity increase
Solution Approach 1:
The session table is divided into multiple partitions based on time ranges. Each partition stores session data for a specific time period, allowing the system to query only relevant partitions rather than scanning all session data. This segmentation reduces computational complexity while maintaining real-time attack identification capability.
Solution Approach 2:
Session data is pre-processed and organized into time-based partitions before querying occurs. The rollup process periodically consolidates historical session data into appropriate partitions in advance, so that when security events need to be correlated, the data is already structured and ready for efficient querying without requiring complex real-time processing.
2Reliability
If all session data is retained and queried to ensure accurate attack detection, then false negatives are reduced, but the time and resources required for data retrieval increase
Solution Approach 1:
By partitioning session data into time-based segments, the system can quickly locate and query only the relevant time period partitions when a security event occurs, rather than scanning all historical session data. This significantly reduces data retrieval time while ensuring all relevant session information is captured for accurate attack detection.
Solution Approach 2:
The rollup process periodically consolidates and reorganizes session data into time-based partitions at scheduled intervals. This periodic reorganization ensures data is efficiently structured for retrieval without requiring continuous complex processing, balancing data retention with retrieval efficiency.
3Measurement precision
If session tables are frequently updated to reflect real-time network state changes, then the accuracy of security event correlation improves, but the system performance and stability may deteriorate
Solution Approach 1:
Instead of continuously updating and reorganizing session data in real-time, the system performs rollup operations at periodic intervals. During these periodic operations, session data is consolidated into time-based partitions. Between rollup operations, the system can efficiently query and correlate security events with session data without the overhead of continuous reorganization, maintaining both accuracy and performance.
Solution Approach 2:
Partitioning session data into time-based segments allows the system to update and manage smaller, manageable portions of data periodically rather than handling the entire dataset continuously. This segmentation reduces the impact of update operations on overall system performance while maintaining correlation accuracy.
4Loss of information
If comprehensive session information is stored to enable detailed attack investigation, then the quality of security analysis improves, but the storage requirements and data management complexity increase
Solution Approach 1:
Session data is segmented into time-based partitions, with each partition containing session information for a specific time period. This segmentation allows the system to retain comprehensive session information across different time ranges while organizing it in a manageable structure. When investigating attacks, the system can access relevant partitions without being overwhelmed by the total volume of data.
Solution Approach 2:
Instead of organizing data by session ID or device and having to filter by time, the system inverts the organization by creating time-based partitions first. Each partition then contains all session data for that time period, making time-based queries efficient while still preserving complete session information for investigation purposes.
Data Source
AI summary
A session table includes one or more records, where each record represents a session. Session record information is stored in various fields, such as key fields, value fields, and timestamp fields. Session information is described as keys and values in order to support query/lookup operations. A session table is associated with a filter, which describes a set of keys that can be used for records in that table. A session table is populated using data contained in security information/events. Rules are created to identify events related to session information, extract the session information, and use the session information to modify a session table. A session table is partitioned so that the number of records in each session table partition is decreased. A session table is processed periodically so that active sessions are moved to the current partition.