Historical Security Data Indexing with Custom Attributes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The exponential growth of raw security signals necessitates more efficient methods for collecting, analyzing, and prioritizing security data to enhance detection and response efficiency in cloud environments, reducing cognitive load, and minimizing incident risks.
Innovation Solution
A system and method for ingesting, indexing, and retrieving structured security data across disparate computing resources, utilizing timestamps, ingestion-attributes, and custom indexing-attributes, with eviction and retrieval mechanisms to manage and prioritize data efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If security data is collected from multiple disparate computing resources, then the quantity of security signals increases, but the complexity of data management and analysis increases
Solution Approach 1:
The patent segments security data into discrete events with specific attributes (timestamps, ingestion-attributes, custom indexing-attributes) that can be independently processed and indexed. This segmentation allows the system to handle large quantities of security signals from disparate sources by breaking them down into manageable, structured units that can be efficiently stored and queried.
Solution Approach 2:
The patent introduces an intermediary indexing layer that sits between raw security data collection and analysis. Custom indexing-attributes act as mediators that bridge disparate data sources and enable unified search capabilities, allowing the system to manage complexity by introducing structured intermediate representations of the security events.
2Loss of information
If all security events are retained indefinitely, then data completeness is improved, but storage requirements and retrieval time increase
Solution Approach 1:
The patent applies preliminary indexing actions to security events at the time of ingestion. By pre-processing and structuring data with timestamps and custom indexing-attributes during ingestion rather than during retrieval, the system prepares data for efficient future queries without requiring indefinite retention of raw data in its original form.
Solution Approach 2:
The patent changes the parameter representation of security data from raw, unstructured logs to structured events with specific parameters (timestamps, ingestion-attributes, custom indexing-attributes). This parameter transformation enables efficient filtering and retrieval based on multiple criteria simultaneously, reducing retrieval time while maintaining data completeness.
3Ease of operation
If custom indexing-attributes are added to each event, then search capability is improved, but data processing complexity increases
Solution Approach 1:
The patent creates a universal event structure with standardized attributes (timestamps, ingestion-attributes, custom indexing-attributes) that serves multiple functions simultaneously. This multi-functional attribute structure enables filtering, sorting, searching, and analysis operations to all work with the same data representation, improving search capability while avoiding the need for separate processing pipelines for different operations.
Data Source
AI summary
A method includes ingesting event data over a network for a plurality of events obtained by disparate computing resources. Each event is associated with a respective timestamp and one or more ingestion-attributes. The method includes identifying whether the corresponding event is associated with any custom indexing-attributes defined by a user. The method also includes indexing the corresponding event into a data store as structured data based on the respective timestamp, the one or more ingestion-attributes, and any identified custom indexing-attributes. The method includes evicting any of the events of the event data in the data store for a period of time that satisfies an eviction time period threshold. The method also includes retrieving the data from the data store that is associated with the time range, the ingestion-attributes, or the one custom indexing-attributes.