Log Data Chunking with Metadata Indexing for SIEM Query Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing security information/event management (SIEM) systems face inefficiencies in storing and querying security information/events, particularly when searching based on attributes other than event receipt time, leading to time-consuming processes.
Innovation Solution
A logging system that includes an event receiver and a storage manager, utilizing a metadata structure for multi-dimensional indexing, allowing efficient storage and querying of log data by processing log data into chunks, compressing them, and using metadata to facilitate fast querying across various event attributes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If events are stored in a single file organized by event receipt time, then writing efficiency is improved (append operations only), but querying efficiency for non-time attributes deteriorates (entire file must be reviewed)
Solution Approach 1:
The patent divides the storage system into multiple files, each dedicated to a specific event source. This segmentation allows the system to maintain append-only writing efficiency while improving query performance for non-time attributes, as queries can be directed to specific source files rather than scanning the entire storage medium.
Solution Approach 2:
The patent introduces a multi-dimensional indexing structure that organizes events not only by time but also by event source and other attributes. This additional dimensional organization enables efficient querying across multiple attributes simultaneously without sacrificing writing efficiency.
2Speed
If events are stored by creating one file per minute of the day, then searching by event receipt time is improved (sequential reading), but storage complexity increases (multiple file management)
Solution Approach 1:
The patent creates a unified storage manager that handles multiple files and indexing dimensions through a single interface. This universal manager abstracts the complexity of multi-file management while maintaining fast search capabilities, allowing users to query by any attribute without dealing with the underlying file structure complexity.
Solution Approach 2:
The patent introduces an intermediary indexing layer between the physical file storage and the query interface. This intermediary structure manages the complexity of multiple files and provides simplified access paths for different query types, reducing the apparent complexity for users while maintaining search speed.
3Productivity
If the oldest file is always deleted first to avoid fragmentation, then deletion efficiency is improved, but flexibility in retention policies deteriorates (cannot selectively retain specific files)
Solution Approach 1:
The patent implements dynamic retention policies that can adapt to different requirements. The system can selectively retain files based on various criteria such as event source, time periods, or security relevance, while maintaining efficient deletion operations through the organized file structure and metadata tracking.
Data Source
AI summary
A logging system includes an event receiver and a storage manager. The receiver receives log data, processes it, and outputs a data “chunk.” The manager receives data chunks and stores them so that they can be queried. The receiver includes buffers that store events and a metadata structure that stores metadata about the contents of the buffers. The metadata includes a unique identifier associated with the receiver, the number of events in the buffers, and, for each “field of interest,” a minimum value and a maximum value that reflect the range of values of that field over all of the events in the buffers. A chunk includes the metadata structure and a compressed version of the contents of the buffers. The metadata structure acts as a search index when querying event data. The logging system can be used in conjunction with a security information/event management (SIEM) system.