Log Query Indexing With Parallel Time-Range Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current log search engines require significant storage space for indexing, leading to high costs as data volume increases, and there is a need for a more efficient and cost-effective solution that reduces storage costs while improving retrieval performance.
Innovation Solution
A log query method that constructs time and data segment indexes dynamically, allowing parallel searching of log content within specified time and data segments, reducing the need for word-based indexing and minimizing storage requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional inverted index construction is used for log search, then retrieval efficiency is improved, but storage cost increases significantly
Solution Approach 1:
The patent segments the traditional inverted index into two separate index structures: a time index that maps time ranges to log file positions, and a keyword index that maps keywords to specific log entries. This segmentation allows the system to avoid storing the complete inverted index for all log data, thereby reducing storage cost while maintaining retrieval efficiency through targeted queries.
Solution Approach 2:
The patent extracts only the essential indexing components needed for log retrieval. Instead of storing the full inverted index with all keyword-document mappings, it extracts and stores only the time index structure and selective keyword indexes, removing redundant data while preserving the core retrieval functionality.
2Measurement precision
If comprehensive indexing is performed on all log data, then query accuracy is improved, but storage space requirements increase
Solution Approach 1:
The patent applies local quality by creating indexes with different levels of detail for different parts of the log data. The time index provides precise temporal localization, while keyword indexes provide selective content localization. This localized indexing approach maintains query accuracy for specific time ranges and keywords without requiring comprehensive indexing of all log data.
Solution Approach 2:
The patent performs preliminary action by pre-computing and storing only the time index structure and essential keyword mappings before queries are executed. This preliminary indexing action prepares the data for efficient retrieval without performing exhaustive indexing on all possible query combinations, thereby reducing storage space while maintaining query accuracy.
3Adaptability or versatility
If word segmentation and processing are performed during data writing, then search capability is improved, but storage cost and processing overhead increase
Solution Approach 1:
The patent extracts only the necessary search components from the comprehensive word segmentation process. Instead of segmenting and indexing all words in log data, it extracts and indexes only time stamps and significant keywords, removing redundant segmentation overhead while preserving essential search capability.
Solution Approach 2:
The patent uses a lightweight, simplified indexing approach that creates smaller, more manageable index structures. The time index and selective keyword indexes are cheaper to store and maintain compared to comprehensive inverted indexes, while still providing adequate search capability for typical log query scenarios.
Data Source
AI summary
A log query method, a medium, and an electronic device are provided. The method includes: receiving a log query request; searching for a target time index in a target time index file and a target log storage file from a time index file constructed based on a log and a log storage file, respectively, and searching for a target query range corresponding to the target time index from the target log storage file; constructing a query plan according to the target query range; and according to the query plan, controlling a worker node to search in parallel for a data segment index matching with a data block specified by an assigned query sub-task from a data segment index file corresponding to the target log storage file, and querying a target log content based on a target data block corresponding to the data segment index.