Log Record Search Estimation Using MinHash and Bit Arrays
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In large data centers, full-text queries on log records consume excessive processing resources and time due to the large number of search iterations, leading to system resource wastage and increased burden on computer systems.
Innovation Solution
A method that generates estimates for the number of occurrences of search query terms in log streams using histogram characterization, MinHash, and bit array techniques, allowing operators to modify queries without running full-text queries on all content, thereby reducing resource consumption and improving search efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If full-text queries are performed on log records to search for terms, then search accuracy is improved, but processing resources and time consumption increase significantly
Solution Approach 1:
The system performs preliminary actions by pre-processing log records to extract and index only the most relevant terms (e.g., status codes, error messages, timestamps) rather than indexing all content. This preliminary term extraction and indexing enables faster search queries while consuming fewer processing resources, as the search only needs to query the pre-indexed terms rather than scanning entire log records.
Solution Approach 2:
The invention extracts only the essential searchable terms from log records and stores them in a specialized index structure. By taking out only the relevant terms (status codes, error messages, key parameters) and separating them from the full log record content, the system enables accurate searches on these extracted terms without requiring full-text processing of all log data, thus reducing processing resource consumption.
2Measurement precision
If full-text queries are performed on log records to search for terms, then search accuracy is improved, but query execution time increases significantly
Solution Approach 1:
The system performs preliminary term extraction and indexing of log records before actual search queries are executed. By pre-processing and organizing the most relevant terms in an optimized index structure, the system enables rapid query execution without needing to scan through entire log records in real-time, significantly reducing query execution time while maintaining search accuracy.
Solution Approach 2:
The invention creates a simplified copy of the search index containing only the most relevant terms and their metadata, rather than copying entire log records. This selective copying approach allows the search system to quickly query and retrieve information from the condensed index, reducing query execution time while preserving the ability to accurately search for specific terms.
3Measurement precision
If iterative full-text queries are performed to refine search results, then search quality is improved, but system burden increases with the number of search iterations
Solution Approach 1:
The system performs preliminary term extraction and indexing of log records before actual search queries are executed. By pre-processing and organizing the most relevant terms in an optimized index structure, the system enables rapid query execution without needing to scan through entire log records in real-time, significantly reducing query execution time while maintaining search accuracy.
Solution Approach 2:
The invention creates a simplified copy of the search index containing only the most relevant terms and their metadata, rather than copying entire log records. This selective copying approach allows the search system to quickly query and retrieve information from the condensed index, reducing query execution time while preserving the ability to accurately search for specific terms.
Data Source
AI summary
A method by a computer includes receiving a search query from a user equipment, where the search query defines a logical combination of terms to be searched within a defined interval of records of a log stream. An estimate is generated for the number of occurrences of the logical combination of terms in the defined interval of records. A message containing the estimate for the number of occurrences of the logical combination of terms in the defined interval of records is communicated toward the user equipment.