Inverted Index Reference Values for Event Record Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current search engines face challenges in efficiently processing large volumes of machine-generated data due to their inability to effectively utilize event-based searching, time-based metadata, and the inflexibility of existing database systems, which discard unused data fields during ingestion, leading to delays and limited retrieval capabilities.
Innovation Solution
The implementation of an inverted index data structure that includes reference values to quickly access and retrieve associated event records, allowing for efficient querying and filtering of raw machine data by leveraging posting values within the inverted index to provide faster access to relevant data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If inverted indexes are used to store all event records for complete search capability, then retrieval flexibility is improved, but storage requirements and computational overhead increase
Solution Approach 1:
The patent extracts only the necessary reference values from event records into the inverted index, rather than storing complete event records. This allows the index to maintain search capability while minimizing storage requirements by including only the minimal data needed for retrieval.
Solution Approach 2:
The patent creates a simplified copy of event record information in the inverted index, storing only reference values that point to the original records. This copy enables fast searching without requiring storage of the full event record data, resolving the contradiction between retrieval flexibility and storage requirements.
2Speed
If inverted indexes include reference values to all event records, then query speed is improved, but computational overhead during index creation increases
Solution Approach 1:
The patent extracts only essential reference values during index creation rather than processing complete event records. This extraction approach reduces computational overhead during index building while maintaining the ability to quickly retrieve records during query operations.
Solution Approach 2:
The patent performs preliminary extraction of reference values during the indexing phase, preparing the inverted index in advance with all necessary reference information. This preliminary action enables fast query speed without requiring complex computational operations during actual search operations.
3Productivity
If ETL systems extract only designated fields during data ingestion, then processing efficiency is improved, but data flexibility is reduced
Solution Approach 1:
The patent extracts reference values from event records during indexing without requiring complete field extraction during data ingestion. This allows efficient processing during ingestion while maintaining the ability to retrieve and analyze any field from the original records later, thus preserving data flexibility.
Solution Approach 2:
The inverted index acts as an intermediary that stores reference values pointing to the original event records. This intermediary structure enables efficient processing during ingestion while maintaining access to complete data fields when needed, resolving the contradiction between processing efficiency and data flexibility.
Data Source
AI summary
Embodiments of the present disclosure provide techniques for performing searches of event records by leveraging reference values in an inverted index. A method of searching comprises accessing a query associated with a first set of event records in a field searchable data store, each event record comprising a time-stamped portion of raw machine data. The method further comprises evaluating the query and generating results for the query by accessing an inverted index, wherein each entry in the inverted index comprises at least one field, a corresponding at least one field value and a reference value that identifies a location in the field searchable data store where an associated event record is stored. The method further comprises performing a search to filter out a second set of event records and retrieving the second set of event records from the field searchable data store using reference values in the inverted index.