Event-Based Data Intake System for Unstructured Machine Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern data centers face challenges in efficiently processing and analyzing vast volumes of machine-generated data due to its unstructured nature, making it difficult to apply semantic meaning and perform effective indexing and searching operations.
Innovation Solution
An event-based data intake and query system with a flexible schema, known as a late-binding schema, is employed to process and store machine data as events with timestamps, allowing for field-searchable data storage and dynamic extraction rules application during search time, enabling efficient retrieval and analysis of machine data from diverse sources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional structured data storage methods are used, then data can be efficiently indexed and searched, but unstructured machine-generated data cannot be effectively processed and analyzed
Solution Approach 1:
The patent changes the fundamental parameter of data structure from structured to unstructured, allowing the system to accept and process machine-generated data in its native format without requiring transformation into predefined schemas, thereby improving adaptability while maintaining processing efficiency through event-based architecture
Solution Approach 2:
The patent introduces an event-based intermediary layer that sits between data ingestion and analysis, where unstructured machine data is converted into standardized events with contextual metadata, enabling both flexible processing of diverse data types and efficient retrieval through structured event schemas
2Quantity of substance
If data is stored in unstructured format, then all raw data can be retained, but indexing and searching operations become difficult and inefficient
Solution Approach 1:
The patent segments unstructured data into discrete events with specific schemas, where each event represents a distinct machine-generated occurrence with defined fields and attributes, enabling both complete data retention and efficient indexing through structured event segmentation
Solution Approach 2:
The patent creates a universal event schema that can represent multiple types of machine-generated data from diverse sources through a common structure, allowing the system to handle varied unstructured data formats while maintaining consistent indexing and searching capabilities across all data types
3Loss of information
If semantic meaning is applied to unstructured data, then data analysis capability improves, but processing complexity increases significantly
Solution Approach 1:
The patent applies preliminary action by automatically extracting and assigning semantic meaning to unstructured machine data at the point of ingestion, converting raw data into events with predefined schemas and contextual metadata before storage, thereby preserving semantic information while avoiding the complexity of post-processing analysis
Solution Approach 2:
The patent implements self-service by enabling the data processing system to automatically interpret and structure unstructured machine data using built-in event schemas and extraction rules, eliminating the need for complex external processing systems while maintaining semantic information integrity
Data Source
AI summary
An actionable event collector in a server cluster receives information specifying an actionable event instance regarding an actionable event occurrence in the server cluster. The actionable event collector transmits a representation of the actionable event instance to an actionable event queue builder. The actionable event queue builder inserts the representation as an entry into an actionable event queue. The event action dispatcher processes the entry from the actionable event queue, wherein processing the entry comprises determining a responsive action for the entry and causing performance of the responsive action.