Log Pre-scanning System for Rapid Symptom Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In large IT environments, managing and finding valuable information from vast volumes of unstructured log data is a time-consuming and difficult task due to the sheer scale of records, making it challenging to identify relevant information efficiently.
Innovation Solution
Implementing a pre-scanning process that identifies symptom matches in logs as they are received or in batches, using pattern-based and comparison-based methods, and storing these matches in a repository for later use in responding to search queries, allowing for quicker identification of logs of interest during scans or searches.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If logs are parsed and stored in a repository for later retrieval, then information accessibility is improved, but the time required to search through tens of billions of records increases significantly
Solution Approach 1:
The patent applies preliminary action by pre-scanning logs and pre-computing symptom matches before actual search queries are executed. The system performs advance processing to identify and store symptom occurrences in a repository, so that when a search query arrives, the results can be quickly retrieved from pre-computed data rather than scanning all logs from scratch. This resolves the contradiction by trading some initial processing time for dramatically reduced query response time.
Solution Approach 2:
The patent creates a copy of log data in a structured repository format that is optimized for querying. Instead of searching through the original unstructured log files, the system maintains a copied and indexed version of the data with pre-identified symptom matches. This copying approach allows fast retrieval of relevant information without requiring access to the full volume of original logs, thus reducing search time while maintaining information accessibility.
2Measurement precision
If all log entries are scanned and processed to ensure complete information retrieval, then measurement completeness is improved, but processing time and computational resources increase
Solution Approach 1:
The patent extracts only the relevant symptom information from the massive log data and stores it in a compact repository. Instead of processing and retrieving all log entries, the system extracts and stores only the symptom matches that are relevant to potential search queries. This extraction approach maintains measurement completeness for the information of interest while dramatically reducing the amount of data that needs to be processed and stored, thereby improving processing efficiency.
Solution Approach 2:
The patent applies partial action by scanning and processing only the portions of log data that contain symptom matches, rather than uniformly processing all log entries. The pre-scan process identifies and flags only the relevant segments of logs that contain symptoms of interest, allowing the system to achieve complete information retrieval for those symptoms without the overhead of processing irrelevant log data. This selective processing maintains completeness for target information while improving overall processing efficiency.
Data Source
AI summary
Techniques to search logs that contain a massive number of entries are disclosed. In various embodiments, logs are pre-scanned incrementally to identify logs of interest. A log is identified as a log of interest based at least in part on a determination that the log satisfies a symptom definition associated with a symptom. A symptom pre-scan result is generated for those logs of interest that satisfy the symptom definition. The symptom pre-scan result is stored in a searchable pre-scan result data store.