Neural Network Field Extraction for Machine Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern data centers face challenges in processing and analyzing large volumes of machine-generated data due to the unstructured nature of the data, which makes it difficult to perform indexing and searching operations, and manual field extraction can be time-consuming and tedious for users.
Innovation Solution
A neural network is used to automatically identify variable text for field extraction recommendations, providing users with automated suggestions for data fields to extract from machine-generated data, facilitating easier data analysis and storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If manual field extraction is used, then users can select fields of interest, but the process is time-consuming and tedious
Solution Approach 1:
The system performs field extraction automatically without requiring user intervention. The neural network analyzes event data and autonomously identifies and extracts relevant fields, eliminating the need for manual selection by users and significantly reducing the time required for field extraction.
Solution Approach 2:
The patent replaces the manual mechanical process of field extraction with an automated neural network system. The neural network uses pattern recognition and machine learning to automatically identify and extract fields from event data, substituting human effort with an intelligent automated system.
2Quantity of substance
If data is pre-processed and extracted before storage, then storage space is reduced, but relevant data may be discarded
Solution Approach 1:
The system performs preliminary analysis of event data using neural networks to identify and extract only the most relevant fields before storage. This preliminary action filters out redundant or irrelevant data while preserving valuable information, optimizing storage space without discarding relevant data.
Solution Approach 2:
The system incorporates feedback mechanisms where users can review and correct automatically extracted fields. This feedback loop ensures that relevant data is not discarded, as users can adjust the extraction results before final storage, maintaining data quality while reducing storage requirements.
3Loss of information
If unstructured data is maintained to reduce processing, then data loss is reduced, but indexing and searching become difficult
Solution Approach 1:
The system segments unstructured event data into structured fields by identifying and extracting relevant information elements. The neural network divides the unstructured data into meaningful fields such as timestamps, hostnames, and event types, enabling efficient indexing and searching while preserving the original data integrity.
Solution Approach 2:
The neural network acts as an intermediary between the unstructured event data and the indexing/searching systems. It translates and transforms the unstructured data into a structured format that can be easily indexed and searched, while maintaining the original unstructured data for reference and analysis.
Data Source
AI summary
Systems and methods include obtaining a set of events, each event in the set of events comprising a time-stamped portion of raw machine data, the raw machine data produced by one or more components within an information technology or security environment and reflects activity within the information technology or security environment. Thereafter, a first neural network is used to automatically identify variable text to extract as a field from the set of events. An indication of the variable text is provided as a field extraction recommendation, for example, to a user device for presentation to a user.