Real-Time Machine Data Search via Query Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional database-oriented search systems are inefficient for real-time searches, causing delays that can be critical in time-sensitive applications, such as security threat detection, and consume excessive processing power when handling large datasets.
Innovation Solution
A system that separates search queries into real-time and historical queries, allowing machine data to be searched and reported in real-time without accessing a database, using an indexing engine to process and store data for immediate results while also storing it in a database for later use.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If conventional database-oriented search systems are used to store and search machine data, then the data can be searched, but significant delays occur between data collection and searching
Solution Approach 1:
The system segments the search process into two distinct paths: real-time searching of recently collected data and historical searching of database-stored data. This segmentation allows the system to provide immediate search results for time-sensitive queries without requiring database access, thereby eliminating the delay between data collection and searching while maintaining the ability to perform historical analysis.
Solution Approach 2:
The patent introduces an intermediary component that buffers recently collected machine data before it is stored in the database. This intermediary allows the system to retrieve and search recent data directly without querying the database, thus acting as a mediator that eliminates the time delay between data collection and searching while preserving the database for historical searches.
2Productivity
If conventional search systems periodically search the database for new machine data, then continuous reporting is achieved, but excessive processing power is consumed
Solution Approach 1:
The system performs preliminary actions by maintaining a buffer of recently collected data in memory before database storage. This preliminary organization of data allows the system to provide continuous reporting by searching the in-memory buffer rather than periodically querying the entire database, thereby achieving continuous monitoring capability while dramatically reducing processing power consumption.
Solution Approach 2:
The patent applies local quality by optimizing the data storage structure for different access patterns. Recently collected data is stored in an easily accessible format in memory for rapid continuous searching, while historical data remains in the database. This local optimization allows continuous reporting operations to access only the necessary recent data with minimal processing, reducing overall energy consumption while maintaining reporting capability.
3Loss of time
If data is continuously written to the database as it is collected, then real-time searchability is improved, but writing efficiency deteriorates
Solution Approach 1:
The system segments the data storage process into two stages: first, data is quickly appended to the buffer in memory with minimal processing overhead; second, data is periodically flushed to the database in bulk operations. This segmentation improves writing efficiency by avoiding frequent database write operations while maintaining real-time searchability through the in-memory buffer that contains recently collected data.
Solution Approach 2:
The patent implements periodic action by flushing data from the in-memory buffer to the database at scheduled intervals rather than continuously. This periodic bulk writing approach significantly improves data writing efficiency by reducing the frequency of database access operations, while the buffer ensures that recently collected data remains searchable in real-time without requiring immediate database persistence.
Data Source
AI summary
A system arranged to search machine data to generate reports in real time. A search query is provided that includes a plurality of search commands. The search query is parsed to form a main search query and a remote search query. Machine data is collected from remote data sources and evaluated against one of the main and remote search queries to generate a set of search results. The main search query is then evaluated against at least a partial set of the search result to generate at least one report regarding the collected machine data. Initially a search window is pre-populated with historical machine data related to the search query. Over time the historical machine data is replaced with the collected machine data.