Accelerated Data Model for Dynamic Field Extraction in Event Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current search engines struggle with efficiently searching large volumes of machine-generated data due to the lack of event-based searching capabilities and the inflexibility of existing database systems, which discard unused data fields, leading to delays and inefficiencies in processing queries.
Innovation Solution
Employing an inverted index and data model that includes structured and unstructured fields, allowing for efficient access and annotation of event records at search time, using a summarization table to expedite query responses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If ETL-based database systems extract and transform data during ingestion, then data can be stored in proper format for querying, but certain data fields are discarded and cannot be retrieved later
Solution Approach 1:
The patent extracts only the necessary structured fields from raw data during ingestion while preserving the complete raw data in its original format. This allows the system to work with formatted data for efficient querying while retaining access to any field that might be needed later, eliminating the information loss problem of traditional ETL systems.
Solution Approach 2:
The system performs preliminary formatting and structuring of data during ingestion without finalizing the extraction process. By maintaining both the structured view and the complete raw data, the system prepares data for efficient querying while leaving the option open to retrieve any field later, thus preventing information loss.
2Productivity
If users pre-specify data fields for extraction at data ingestion time, then data can be processed efficiently, but the system becomes inflexible and cannot retrieve fields not designated for extraction
Solution Approach 1:
The patent implements a dynamic field extraction system where users can specify fields for efficient processing when needed, while the system maintains the capability to extract any field from the preserved raw data. This dynamic approach allows the system to switch between pre-specified field extraction and ad-hoc field retrieval, providing both efficiency and flexibility.
Solution Approach 2:
The system serves multiple functions: it efficiently processes data using pre-specified structured fields for common queries, while simultaneously preserving the ability to extract any field from the complete raw data for ad-hoc analysis. This multi-functionality resolves the contradiction between efficiency and flexibility.
3Reliability
If operations examine and scan individual data records, then complete data analysis is possible, but processing becomes prohibitively expensive as data set size grows
Solution Approach 1:
The patent segments data into structured fields that are pre-processed and indexed during ingestion, and complete raw data that is preserved but not fully scanned. This segmentation allows the system to efficiently query only the necessary structured portions for most operations, reducing processing costs while maintaining the ability to access complete data records when needed.
Solution Approach 2:
The system performs preliminary processing and structuring of data during ingestion, creating an optimized view for efficient querying. This preliminary action reduces the need for expensive full scans of individual records during analysis operations, while the complete raw data remains available for comprehensive analysis when required.
4Quantity of substance
If storage capacity becomes cheaper, then there are fewer incentives to discard unused portions of raw data records, but searching large sets of data becomes more difficult
Solution Approach 1:
The patent segments the data storage into two parts: structured fields that are pre-processed and indexed for efficient searching, and complete raw data that is preserved in its original form. This segmentation allows the system to maintain large volumes of data while providing efficient search capabilities through the structured portion, reducing the complexity of searching large data sets.
Solution Approach 2:
The structured fields act as an intermediary between the complete raw data and the search query. This intermediary layer provides an efficient interface for searching large data sets, translating complex search requirements into efficient queries against the structured portion while maintaining access to the complete data volume.
Data Source
AI summary
Embodiments of the present disclosure provide techniques for emitting structured and dynamic fields from an accelerated data model. The method comprises evaluating a query to search a data model, wherein the data model is defined by a set of events and at least one structured field from fields associated with the set of events. Each event comprises a time-stamped portion of raw machine data and is stored in a field searchable data store. A summarization table is associated with the data model and comprises a plurality of entries comprising reference values, wherein a respective summarization table entry comprises: the at least one structured field; a respective field value; and a reference value. The method further comprises accessing the set of events from the field searchable data store using the reference values in the summarization table and annotating the set of events with the at least one structured field and with at least one dynamic field from the fields associated with the set of events, wherein the at least one dynamic field is not defined in the data model.