Minimally Processed Data Intake for Cyber-Attack Identification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing volume of machine-generated data from diverse sources poses challenges in analysis and storage, as traditional systems often discard minimally processed data, limiting flexibility and efficiency in data retrieval and security threat detection.
Innovation Solution
The SPLUNK ENTERPRISE system employs a data intake and query system that stores minimally processed machine data for later analysis, using a late-binding schema to extract relevant information at search time, facilitating flexible data exploration and security threat detection through pre-specified schemas and correlation searches.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional systems discard minimally processed data, then storage costs are reduced, but data retrieval flexibility and security threat detection capability are limited
Solution Approach 1:
The system performs preliminary actions by storing minimally processed data in an optimized format with metadata tags before detailed analysis is needed. This allows the data to be readily available for future security threat detection and analysis without requiring reprocessing, thus improving adaptability while managing storage efficiently
Solution Approach 2:
The system changes the parameter of data storage by maintaining data in a minimally processed state with associated metadata rather than fully processing or discarding it. This parameter change enables flexible retrieval and analysis while optimizing storage space, resolving the contradiction between storage volume and data retrieval flexibility
2Reliability
If all machine data is stored for later analysis, then security threat detection capability is improved, but storage requirements and system complexity increase
Solution Approach 1:
The system extracts only the essential metadata and key characteristics from machine data during initial ingestion, storing these extracted features alongside the minimally processed data. This extraction approach enables effective security threat detection without requiring storage of all raw data details, thus improving reliability while managing system complexity
Solution Approach 2:
The system performs preliminary classification and tagging of data with security-relevant metadata during ingestion. This preliminary action enables efficient security threat detection later without requiring complex real-time processing of all stored data, resolving the contradiction between detection capability and system complexity
3Adaptability or versatility
If minimally processed data is maintained for later retrieval, then analysis flexibility is improved, but data processing time at ingestion is reduced
Solution Approach 1:
The system extracts only the necessary metadata and key identifiers from incoming machine data during ingestion, rather than performing full processing. This extraction approach maintains analysis flexibility for later retrieval while preserving data ingestion efficiency by minimizing processing overhead at ingestion time
Solution Approach 2:
The system performs partial processing of data at ingestion, maintaining it in a minimally processed state with essential metadata. This partial action approach provides sufficient analysis flexibility for most use cases while maintaining high ingestion efficiency, resolving the contradiction between flexibility and productivity
Data Source
AI summary
In a method, a plurality of events is accessed, wherein an event of the plurality of events includes a portion of raw-machine data from a data source of a plurality of data sources. For at least one event of the plurality of events, a transaction phase of a computer security transaction is correlated with the at least one event based at least in part on a data source associated with the at least one event. The transaction phase of the at least one event is correlated with a particular asset of a plurality of assets.