Structured Security Event Indexing for Historical Search at Scale
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The exponential growth of raw security signals necessitates more efficient methods for ingesting, indexing, and retrieving historical security data across disparate computing resources to enhance detection and response efficiency in cloud environments.
Innovation Solution
A system for ingesting event data with timestamps and attributes, indexing it as structured data based on user-defined criteria, and managing its retention and retrieval through a distributed storage system, enabling efficient and prioritized data access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If security data is collected from disparate computing resources, then the quantity of security signals increases, but the complexity of storage and retrieval increases
Solution Approach 1:
The system segments security data into discrete events with specific attributes (timestamp, event type, source, severity) and organizes them into structured collections. Each event is independently indexable and retrievable, allowing the system to manage large quantities of security signals without proportionally increasing retrieval complexity.
Solution Approach 2:
The patent introduces multiple dimensional indexing beyond simple chronological ordering, including event type, source, severity level, and custom attributes. This multi-dimensional organization allows efficient retrieval across different security analysis needs without requiring complex query processing, resolving the contradiction between data quantity and retrieval complexity.
2Loss of information
If all security events are retained indefinitely, then data completeness is improved, but storage resource consumption increases
Solution Approach 1:
The system implements periodic retention policies where events are retained for specified time periods based on their attributes. Events can be automatically archived or deleted after meeting retention criteria, while still allowing retrieval within the retention window. This periodic management maintains data completeness for active security analysis while controlling long-term storage consumption.
3Adaptability or versatility
If custom indexing attributes are defined by users, then data retrieval flexibility is improved, but system configuration complexity increases
Solution Approach 1:
The system provides a universal indexing framework where users can define custom attributes that work across all security events and retrieval operations. The same indexing mechanism handles both standard attributes (timestamp, event type) and custom user-defined attributes uniformly, allowing retrieval flexibility without requiring separate configuration systems for different attribute types.
Data Source
AI summary
A method includes ingesting event data over a network for a plurality of events obtained by disparate computing resources. Each event is associated with a respective timestamp and one or more ingestion-attributes. The method includes identifying whether the corresponding event is associated with any custom indexing-attributes defined by a user. The method also includes indexing the corresponding event into a data store as structured data based on the respective timestamp, the one or more ingestion-attributes, and any identified custom indexing-attributes. The method includes evicting any of the events of the event data in the data store for a period of time that satisfies an eviction time period threshold. The method also includes retrieving the data from the data store that is associated with the time range, the ingestion-attributes, or the one custom indexing-attributes.