Field Discovery for Unstructured Data Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern data centers face challenges in processing and analyzing large volumes of machine-generated data due to its unstructured nature, making it difficult to apply semantic meaning and perform effective indexing and searching operations.
Innovation Solution
The system discovers fields within the data returned from an initial search query, allowing users to further filter results through a graphical user interface (GUI) by selecting and applying criteria to specific fields, using extraction rules to identify and extract relevant values, and generating a data model for enhanced search and visualization capabilities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional indexing and searching operations are applied to unstructured machine-generated data, then the data can be processed and retrieved, but the difficulty of applying semantic meaning to unstructured data makes effective indexing and searching challenging
Solution Approach 1:
The system automatically discovers fields and generates data models without requiring manual configuration or preprocessing of unstructured data. The field discovery mechanism autonomously analyzes the data structure, identifies relevant fields, and creates searchable indexes, enabling the system to serve itself in transforming unstructured data into a searchable format.
Solution Approach 2:
The system performs preliminary field discovery and data model generation before actual search operations. By pre-processing the unstructured data to identify and structure relevant fields in advance, the system prepares the data for efficient searching and indexing without requiring manual intervention during the search process.
2Measurement precision
If users manually analyze and filter large volumes of heterogeneous data, then accurate results can be obtained, but the time and effort required increases significantly
Solution Approach 1:
The system automatically performs field discovery, data model generation, and search result filtering without requiring manual user intervention. The automated mechanisms analyze the data structure, identify relevant fields, and filter results based on discovered fields, enabling the system to serve itself in delivering accurate search results efficiently.
Solution Approach 2:
The system replaces manual mechanical analysis and filtering operations with automated computational processes. Field discovery algorithms and data model generation mechanisms substitute for manual data examination, automatically identifying relevant fields and filtering results without requiring users to manually analyze each data element.
3Adaptability or versatility
If comprehensive field discovery is performed on all returned data, then more filtering options are available, but the complexity of the system increases
Solution Approach 1:
The system extracts and identifies only the relevant fields from the returned data through automated field discovery. Instead of processing or displaying all possible data elements, the mechanism selectively extracts meaningful fields that can be used for filtering, reducing the effective complexity while maintaining comprehensive filtering capabilities for relevant parameters.
Solution Approach 2:
The system applies field discovery and data modeling locally to the specific data returned from search operations, rather than attempting to process all possible data uniformly. The field discovery mechanism adapts to the characteristics of the returned data and identifies fields specific to that data set, providing tailored filtering options without requiring a complex universal processing framework.
Data Source
AI summary
Fields may be discovered in events that are returned in response to an initial search. The events may comprise portions of raw data. Furthermore, the fields may be defined by extraction rules for extracting values from corresponding portions of raw data. The displaying of a graphical user interface (GUI) may be caused where the GUI enables a user to select or enter criteria for a subset of the discovered fields without entering a search query in a search bar. At least one criterion for at least one field from the subset of the discovered fields may be received through a portion of the GUI that does not include a search bar for entering a search query. The events returned in response to the initial search query may be caused to be filtered based on the received criterion.