Search Table Architecture for Faster Continuous Event Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Database performance suffers when real-time search queries are continually run against a large events table as new log events are loaded, impeding the ability to quickly identify potential anomalies.
Innovation Solution
Implement a database architecture that includes a search table for each search query, storing only matching events, and a search job to continually populate this table from the events table, thereby reducing the load on the events table.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If search queries are continually run against the events table to retrieve matching events in real-time, then the ability to detect anomalies is improved, but database performance deteriorates due to increased query load on the large events table
Solution Approach 1:
The patent segments the events table by creating separate search tables for different search queries. Each search table stores only the matching events for its specific query criteria, dividing the large events table into smaller, query-specific subsets. This segmentation reduces the data volume processed during each search operation, thereby improving database performance while maintaining anomaly detection capability.
Solution Approach 2:
The patent implements preliminary action by pre-filtering and storing matching events in separate search tables before they are needed for anomaly detection. The search tables are populated in advance or updated continuously with events matching specific criteria, so that when anomaly detection is required, the data is already organized and ready for quick retrieval without needing to scan the entire events table.
2Adaptability or versatility
If the events table is continuously updated with new log events, then real-time monitoring capability is improved, but query response time deteriorates due to the growing size of the table
Solution Approach 1:
The patent segments the continuously growing events table into multiple smaller search tables, each dedicated to specific search criteria. This segmentation ensures that query response time remains fast even as the events table grows, because queries only need to access the relevant small search table rather than scanning the entire large table.
Solution Approach 2:
The patent extracts matching events from the main events table and stores them in separate search tables. This extraction removes the burden of searching through the entire events table during queries, as the matching events are already pre-organized in the search tables, thereby maintaining fast query response times despite continuous table updates.
3Productivity
If a search table is created for each search query to store only matching events, then query performance is improved, but system complexity increases due to multiple tables and synchronization mechanisms
Solution Approach 1:
The patent implements universality by creating a single search table structure that can serve multiple search queries with different criteria. Instead of creating completely separate tables for each query type, the system uses a unified search table design that can be configured to handle various search scenarios, reducing overall system complexity while maintaining high query performance.
Data Source
AI summary
A search table is instantiated for a search query for which matching events of the events stored in an events table are to be continually provided as new events are continually loaded into the events table. The matching events satisfy the search query. The search table is to store the matching events. A search job for the search query is generated. The search job is to be continually (i.e., periodically) run to retrieve the matching events stored in the events table that are not already stored in the search table and to insert the retrieved matching events in the search table. The search job is therefore continually (i.e., periodically) run, such that the matching events are continually (i.e., periodically) provided from the search table and not from the events table.