Log Indexing Service for Large-Scale Web Services
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale Web services generate enormous volumes of log files, making it time-consuming for administrators to locate specific data, especially when dealing with thousands of service hosts that produce several hundred gigabytes of logs per hour.
Innovation Solution
An indexing service is implemented to periodically retrieve and search log files for specified key values, storing the identified data in a log index, allowing quick access to relevant information through a user interface or API, enabling real-time or near-real-time data retrieval and storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If log files are stored without indexing, then storage capacity is utilized efficiently, but data retrieval time increases significantly
Solution Approach 1:
The patent implements preliminary action by creating indexes in advance of actual data queries. The indexing service continuously generates and maintains indexes of log data as it is produced, so that when administrators need to search for specific information, the data is already organized and searchable, dramatically reducing retrieval time without requiring full log file scans
Solution Approach 2:
The patent introduces an intermediary indexing service that sits between the log generation systems and the search functionality. This intermediary component preprocesses log data, extracts key information, and maintains separate index structures that enable rapid querying without requiring administrators to search through raw log files directly
2Speed
If all log data is indexed, then search speed improves, but storage requirements increase
Solution Approach 1:
The patent applies the extraction principle by selectively pulling out only the most search-relevant information from comprehensive log files. The indexing service extracts key fields such as timestamps, error codes, request IDs, and other queryable attributes, creating compact index structures that contain only the essential searchable data rather than duplicating entire log files
Solution Approach 2:
The patent implements local quality by creating different types of indexes with different levels of detail and compression for different portions of the log data. Frequently queried fields receive more optimized indexing treatment, while less important data uses more compact representations, optimizing the balance between search speed and storage efficiency across different regions of the index structure
3Ease of operation
If real-time indexing is implemented, then data accessibility improves, but processing load increases
Solution Approach 1:
The patent implements periodic action by updating indexes at scheduled intervals rather than continuously in real-time. The indexing service processes log data in batches or at fixed time intervals, which reduces the instantaneous processing load while still maintaining relatively current search capabilities. This approach balances data accessibility with acceptable processing resource consumption
Solution Approach 2:
The patent applies partial action by indexing only a subset of log data at any given time, focusing on the most recently generated logs or the most frequently accessed portions. This allows the system to provide useful search functionality with reduced processing requirements, rather than attempting to index and maintain every single log entry at maximum freshness
Data Source
AI summary
A large-scale service utilizes service hosts that generate log files describing aspects of their operation. An indexing service obtains the log files from a log store and/or obtains log data directly from the service hosts. The indexing service also searches the log files for key values and other associated values. The indexing service utilizes a database service to store located key values and the other values in a log index. A user interface component provides functionality for searching the log index for key values and displaying the other values associated with the key values. An application programming interface (API) might also be utilized to query the log index for key values and to thereby obtain the other associated values.