Row and Column Data Chunks for SIEM Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing security information/event management (SIEM) systems face inefficiencies in storing and querying security information/events, particularly when searching based on attributes other than event receipt time, leading to time-consuming processes and storage fragmentation.
Innovation Solution
A logging system that includes an event receiver and a storage manager, utilizing column-based data chunks with metadata structures to efficiently store and query log data across various event attributes, supporting efficient querying and storage management through the use of column-based and row-based chunk architectures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If events are stored in a single sequential file based on event receipt time, then writing efficiency is improved through append operations, but querying efficiency for non-time attributes deteriorates requiring full file review
Solution Approach 1:
The patent segments the storage system into multiple files organized by time intervals (e.g., one file per minute of the day) and further segments each file into chunks. This segmentation allows efficient append operations within chunks while enabling indexed access across multiple files and chunks, resolving the contradiction between sequential write efficiency and query efficiency for non-time attributes.
Solution Approach 2:
The patent introduces metadata structures as intermediaries that store indexing information about event attributes (such as event source, type, and other non-time attributes) alongside the actual event data. These metadata structures enable efficient querying by providing direct access paths to events based on various attributes without requiring full file reviews, while maintaining the sequential append structure for efficient writing.
2Quantity of substance
If events are stored in a single sequential file, then storage space utilization is improved, but storage fragmentation occurs when deleting oldest event data
Solution Approach 1:
By segmenting storage into multiple time-based files and chunk-based units, the patent enables independent management of storage segments. When oldest events need to be deleted, entire chunks or files can be removed without affecting other segments, preventing storage fragmentation while maintaining efficient space utilization through systematic organization.
Solution Approach 2:
The patent implements a systematic approach where oldest chunks or files are discarded in an organized manner based on retention policies. This allows efficient reclamation of storage space without causing fragmentation, as the segmented structure enables clean removal of entire units rather than partial deletions within files.
3Speed
If multiple files are created for different time intervals, then querying efficiency for time-based attributes is improved, but device complexity increases
Solution Approach 1:
The patent creates a universal metadata structure and chunk format that works across all time-based files. The same metadata schema, chunk organization, and query mechanisms are applied uniformly regardless of which time interval or file is being accessed. This multi-functional approach enables efficient time-based querying while managing complexity through consistent, reusable patterns across the distributed file system.
Data Source
AI summary
In some examples, a set of events is received. A row-based chunk includes the set of events and metadata about the set of events is generated, and a column-based chunk that includes metadata about the set of events and, for each event in the set of events, a value of a first field of the multiple fields. The metadata about the set of events includes at least one of a minimum value or a maximum value of the first field over the events in the set of events.