Indexing Distributed Log Data for Rapid Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional mechanisms for searching logging information in massively parallel software application environments are tedious and time-consuming, hindering troubleshooting and maintenance due to the need to search across multiple host computers and files, which results in high computational complexity and slow performance.
Innovation Solution
The system categorizes logging information using unique identifiers and metadata, allowing for centralized storage and efficient search of key-value pairs, reducing search complexity from O(n) to O(1) and enabling rapid access to specific logging information statements, even in large datasets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional search mechanisms are used to search logging information across multiple host computers and files, then the search can be performed on distributed log data, but the search process is time-consuming and computationally complex
Solution Approach 1:
The patent applies preliminary action by pre-processing logging information during the logging phase. A logging information processing module extracts key-value pairs from log statements and stores them in a specialized data structure (hash table or inverted index) before search operations are needed. This pre-processing creates an indexed representation of the log data that enables rapid retrieval without scanning entire log files during search operations.
Solution Approach 2:
The patent introduces an intermediary component - a specialized search module with optimized data structures (hash tables, inverted indexes) that mediates between the distributed log files and the search query. This intermediary layer transforms the search problem from a full-text scan across multiple files into a direct key-based lookup operation, dramatically reducing search time while maintaining completeness.
2Reliability
If conventional search mechanisms are used to search logging information, then the search can cover all log files, but the computational complexity is high
Solution Approach 1:
The patent changes the parameter representation of log data by transforming unstructured log statements into structured key-value pairs during preprocessing. This parameter transformation enables the use of efficient data structures (hash tables with O(1) lookup, inverted indexes) that reduce computational complexity from linear O(n) scanning to constant or logarithmic time operations.
Solution Approach 2:
The patent substitutes the mechanical scanning process (sequential reading and pattern matching across log files) with a computational lookup mechanism using hash tables and inverted indexes. This replacement changes the fundamental operation from mechanical text processing to efficient computational key-based retrieval, significantly reducing computational complexity.
3Productivity
If logging information is retained in files across multiple host computers, then the log data can be collected from distributed systems, but access to specific portions requires searching across multiple files
Solution Approach 1:
The patent applies segmentation by dividing the logging information into structured key-value pairs during preprocessing. Each log statement is segmented into discrete key-value components that can be independently indexed and searched. This segmentation enables the search module to access specific portions of log data by key without needing to scan entire files or coordinate across multiple host computers during the search operation.
Solution Approach 2:
The patent performs preliminary action by pre-extracting and pre-indexing key-value pairs from distributed log files during the logging phase. This pre-processing creates a centralized or distributed index structure that enables rapid access to specific log information by key, eliminating the need to search across multiple files during actual search operations.
Data Source
AI summary
Architectures and techniques to store and search logging information are provided. In some embodiments, statements included in the logging information can be categorized according to respective unique identifiers. A category of statements can be retained in one or more files within a directory associated with a unique identifiers representative of the category. The directory can be included in a distributed storage system. In addition, metadata including the unique identifier and a defined searchable field or another defined discrete amount of information pertaining to a statement associated with the unique identifier can provide an indexation of an available category of statements. In addition or in other embodiments, an interface unit can permit querying the logging information via programmatic queries or other queries received via a user interface. The query can include a desired search pattern and a response to the query can include a list of one or more unique identifiers indicative of respective statements matching the search pattern. Logging information statements sorted according a defined criterion can be provided in response to the selection.