Temporal Search Term Recommendations for Machine Data Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Analyzing and searching massive quantities of machine-generated data from diverse sources is time-consuming and inefficient due to the varied formats and types of data, which existing systems struggle to handle effectively.
Innovation Solution
Implementing a data intake and query system with a late-binding schema that processes and indexes data at search time, using extraction rules to derive fields dynamically and facilitate flexible schema development, enabling analysis of minimally processed machine data across heterogeneous sources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is processed and indexed at ingestion time with predefined schemas, then data organization and retrieval speed are improved, but flexibility in handling diverse data formats and schema adaptation deteriorate
Solution Approach 1:
The patent implements a late-binding schema approach where field extraction rules are applied dynamically at search time rather than being fixed at ingestion time. This allows the schema to adapt flexibly to diverse data formats while maintaining efficient retrieval through indexed data structures. The system dynamically determines field extractions based on extraction rules evaluated at query time, resolving the contradiction between speed and adaptability.
2Reliability
If extensive data processing and transformation are performed at ingestion time, then data quality and consistency are improved, but processing time and computational resources deteriorate
Solution Approach 1:
The patent performs preliminary indexing of raw data at ingestion time without extensive transformation, storing data in an optimized indexed format for fast retrieval. Field extractions and transformations are deferred to search time when actually needed. This preliminary action maintains data quality through proper indexing while avoiding unnecessary processing time delays.
Solution Approach 2:
The system performs field extractions on-demand at search time based on extraction rules, allowing each query to self-determine the necessary field extractions rather than pre-processing all possible extractions. This self-service approach ensures data quality is maintained only for the specific fields needed, reducing overall processing time and computational resource consumption.
3Productivity
If minimal data processing is performed at ingestion time, then processing speed and resource efficiency are improved, but data organization and analysis capability deteriorate
Solution Approach 1:
The patent performs preliminary indexing of data at ingestion time, organizing raw data into efficient data structures without extensive transformation. This preliminary action maintains high ingestion speed while preparing data for fast retrieval and analysis at search time through the indexed format and associated extraction rules.
Solution Approach 2:
The system dynamically applies field extraction rules at search time based on the specific query requirements, enabling flexible data analysis capability without pre-processing. The extraction rules are evaluated dynamically during search operations, allowing the system to maintain high ingestion productivity while providing comprehensive data analysis capabilities when needed.
Data Source
AI summary
Embodiments of the present invention are directed to facilitating search input recommendations. In accordance with aspects of the present disclosure, a set of events determined from raw machine data is obtained. The events are analyzed to generate a temporal map associated with the set of events. Generally, the temporal map associates candidate terms with temporally related terms that occur within a period of time corresponding with the candidate terms. A search term input into a search field is received. Based on the input search term, the temporal map is used to identify one or more temporally related term recommendations.