Summarization Tables for Event Record Query Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current search engines face challenges in efficiently searching large volumes of machine-generated data, particularly in handling time-stamped event records, as they often rely on keyword searches and lack effective event-based searching mechanisms, leading to expensive operations when dealing with growing data sets.
Innovation Solution
A system that generates summarization information and uses summarization tables to optimize queries, allowing for event-based searching without the need to scan individual event records, by indexing and storing event records in a way that enables fast retrieval and analysis, using configuration information and extraction rules to identify and extract relevant data fields.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If keyword-based search is used to search large volumes of machine-generated data, then the search can be performed with simple mechanisms, but the search effectiveness and event-based retrieval capability deteriorate
Solution Approach 1:
The patent introduces summarization tables as an intermediary structure between raw event records and search queries. These tables pre-compute and store aggregated statistics (counts, sums, averages) for various fields, enabling event-based searches without scanning individual records. The summarization table acts as a mediator that translates complex event-based search requirements into efficient aggregate queries.
Solution Approach 2:
The system performs preliminary actions by pre-computing and storing summarization information before searches are executed. Summarization tables are built in advance with aggregated data from event records, including field statistics and event counts. This preliminary computation enables fast event-based searching without needing to re-process individual records during query execution.
2Loss of information
If individual event records are scanned to generate statistics and metrics, then complete data analysis is achieved, but computational cost increases prohibitively as data volume grows
Solution Approach 1:
The patent extracts only the essential statistical information needed for analysis from the full event records. Summarization tables store pre-computed aggregates (counts, sums, averages, min/max values) for specified fields, separating the essential analytical data from the complete raw records. This extraction allows statistics generation without processing every individual event record.
Solution Approach 2:
The system creates simplified copies of event data in the form of summarization tables. These tables contain replicated aggregate statistics that mirror the essential characteristics of the full event dataset without containing the complete raw records. Queries operate on these copied aggregate representations, dramatically reducing computational requirements while preserving analytical capabilities.
3Speed
If indexing is applied to enable quick identification of event records, then retrieval speed improves, but operations that examine individual records become prohibitively expensive
Solution Approach 1:
The patent adds another dimension to the data structure by creating summarization tables that exist alongside indexed event records. These tables provide a different view of the data at the aggregate level, allowing queries to operate in this higher-dimensional space of pre-computed statistics rather than examining individual records, thus avoiding the complexity of detailed record examination.
4Productivity
If summarization tables are used to answer queries without scanning individual records, then computational cost is reduced, but the ability to handle arbitrary queries may be limited
Solution Approach 1:
The summarization tables are designed with multi-functionality to handle various types of queries. They store aggregate statistics for multiple fields and support different query operations (counts, sums, averages, filtering). The tables can serve multiple query purposes simultaneously, making the system both efficient and adaptable to different analytical needs without requiring separate processing for each query type.
Data Source
AI summary
Embodiments are directed are towards a method for generating a query response, which comprises creating two or more partitions of event records from raw data stored in a data store, wherein each event record in the two or more partitions of event records includes a portion of the raw data and is associated with a time stamp derived from the raw data. The method also comprises generating a summarization table for each partition of the two or more partitions that: (a) identifies a field value comprising a value that corresponds to an associated field extracted from a respective event record; and (b) for the field value, includes a posting value to the respective event record within a respective partition. The method further comprises generating partial results for a received query using summarization tables in the partitions and generating a response to the query by combining the partial results.