Column-Based Event Data Search With Contextual Query Options
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern data centers face challenges in processing and indexing large volumes of heterogeneous performance data due to its unstructured nature, leading to difficulties in semantic meaning application and effective search operations, with traditional database systems often discarding valuable data during pre-processing.
Innovation Solution
The SPLUNK® ENTERPRISE system employs a late-binding schema and event-based processing to store and analyze performance data, allowing flexible schema development at search time, enabling column-based table manipulation and contextually relevant options for user interaction, and utilizing acceleration techniques like map-reduce computations and keyword indexing to enhance query efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional database systems perform pre-processing on performance data, then storage space is reduced, but valuable data is discarded
Solution Approach 1:
The system recovers discarded data by maintaining original performance data in its unstructured form while creating structured representations only when needed for analysis. The search interface can access both the original data and processed representations, allowing recovery of valuable information that would otherwise be lost during pre-processing.
Solution Approach 2:
The system segments data processing into multiple representations: original unstructured data is preserved, while structured data is generated on-demand through schema binding. This segmentation allows the system to maintain both compact storage (through selective structuring) and data recovery (through access to original forms).
2Loss of information
If unstructured data is maintained to reduce processing, then data loss is reduced, but indexing and searching operations become challenging
Solution Approach 1:
The system dynamically transforms unstructured data into structured representations at search time through schema binding. The schema is dynamically selected and applied based on the search query context, allowing the system to maintain unstructured data for storage while creating structured indices on-demand for efficient searching and indexing operations.
Solution Approach 2:
The schema acts as an intermediary between the unstructured data and the search operations. It mediates by providing a structured framework that can be applied to unstructured data, enabling indexing and searching without permanently altering the original data format.
3Productivity
If large volumes of machine-generated data are processed, then analysis capability is improved, but processing complexity increases
Solution Approach 1:
The system segments the data processing workload by handling data at different levels of abstraction. The search interface operates on high-level structured representations while the underlying system manages low-level unstructured data. This segmentation reduces processing complexity at the user interface level while maintaining analysis capability through the structured representations.
Solution Approach 2:
The system creates a copy of the data in a structured format for analysis and search operations, while the original unstructured data remains intact. This copying approach allows complex analysis operations on structured data without increasing the complexity of storing and managing the original data volumes.
4Loss of information
If search results are provided in large quantities, then search completeness is improved, but user interpretation difficulty increases
Solution Approach 1:
The system applies local quality by providing structured, contextually relevant information at the point of search results. Each search result is enriched with schema-based structure and contextual metadata, making the information more interpretable. This local structuring improves user interpretation without reducing the quantity of search results provided.
Data Source
AI summary
A search interface is displayed in a table format that includes a plurality of columns, each column including data items of an event attribute, the data items being of a set of events, each column being selectable by a user, and a plurality of rows forming cells with the one or more columns, each cell comprising one or more of the data items of the event attribute of a corresponding column. Based on the user selecting one or more of the columns, a list of options is displayed corresponding to the selected one or more columns, and one or more commands are added to a search query that corresponds to the set of events. The one or more commands are based on at least an option that is selected from the list of options and the event attribute of each of the selected one or more columns.