Event-Based Data Intake System for Unstructured Machine Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Analyzing and searching massive quantities of machine data from diverse sources in data centers is challenging due to the unstructured nature of the data and the difficulty in applying semantic meaning, leading to inefficiencies in processing and indexing.
Innovation Solution
An event-based data intake and query system uses a flexible schema to process and store machine data as events with timestamps, enabling field-searchability and late-binding schema for extraction rules, allowing for real-time analysis and querying across disparate data sources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional indexing and searching methods are used on unstructured machine data, then data can be stored, but searching and analysis become inefficient and difficult
Solution Approach 1:
The patent applies preliminary action by automatically generating extraction rules and applying them during data ingestion, before search queries are executed. The system pre-processes unstructured machine data by extracting relevant fields and organizing them into a structured format with timestamps and event types, making the data readily searchable without requiring complex query processing later.
Solution Approach 2:
The patent introduces an intermediary layer between raw unstructured machine data and search queries. This intermediary consists of automatically generated extraction rules that transform unstructured data into structured events with standardized fields. This intermediary structure enables efficient searching by providing a consistent interface between diverse data sources and search operations.
2Adaptability or versatility
If a rigid schema is used for data processing, then data structure is maintained, but flexibility in adapting to diverse data sources is reduced
Solution Approach 1:
The patent implements dynamics by making the data schema adaptable and evolving rather than fixed. The system automatically generates extraction rules based on incoming data patterns and refines these rules over time based on search query feedback. This dynamic approach allows the schema to adapt to diverse data sources automatically, reducing the need for manual schema development and maintenance while handling varying data structures from different machine sources.
3Manufacturing precision
If manual schema development is performed for each data source, then data structure is optimized, but time and resources are significantly consumed
Solution Approach 1:
The patent applies self-service by enabling the system to automatically generate and refine its own extraction rules without manual intervention. The system analyzes incoming machine data patterns, automatically creates appropriate extraction rules, and continues to refine these rules based on search query performance and feedback. This self-service capability eliminates the need for manual schema development for each new data source, significantly reducing time and resource consumption while maintaining optimized data structures.
Data Source
AI summary
Machine data of an operating environment is conveyed by a network to a data intake and query system (DIQS) which reflects the machine data as timestamped entries of a field-searchable datastore. Monitoring functionality may search the machine data to identify notable event instances. A notable event processing system correlates the notable event instance to one or more triaging models which are executed against the notable event to produce a modeled result. Information of the received notable event and the modeled results are combined into an enhanced representation of a notable event instance. The enhanced representation conditions downstream processing to automatically perform or assist triaging of notable event instances to optimize application of computing resources to highest priority conditions in the operating environment.