Log Intelligence Metric Caching for Real-Time Dashboard Performance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Log intelligence systems face inefficiencies in generating real-time dashboard analytics and alerts due to the need to query large volumes of indexed logs, especially when data is distributed across multiple shards, leading to slow performance and increased data retrieval times.
Innovation Solution
Implementing a method to cache only the metrics needed for real-time dashboard services during the log intake process by creating a dedicated data store that contains a subset of log data, which is continuously updated with current operational system data, and using a constraint plane to extract and store relevant metrics based on predefined queries, thereby reducing the amount of data stored and improving access times.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the system queries indexed logs stored in a text indexing system to provide real-time reporting services, then the system can provide user-facing functionalities such as real-time queries, alerts, dashboards and analytics, but the performance becomes inefficient and data retrieval times increase when logs are distributed across multiple shards
Solution Approach 1:
The patent segments the data storage into two separate planes: a first data plane for storing complete log data and a second data plane for storing only the metrics needed for real-time reporting. This segmentation allows the system to maintain comprehensive logging capabilities while optimizing query performance by storing only necessary data in the second plane, resolving the contradiction between versatility and productivity.
Solution Approach 2:
The patent extracts only the necessary metrics from the complete log data and stores them separately in the second data plane. This extraction process eliminates the need to query through all distributed log shards for real-time reporting, significantly improving query performance while maintaining the ability to access complete logs when needed.
2Loss of information
If the system stores complete log data in a text indexing system, then the system can provide comprehensive data for analysis, but the amount of data stored increases and access times for real-time reporting increase
Solution Approach 1:
The patent divides the data storage into two segmented planes: the first data plane stores complete log data for comprehensive analysis, while the second data plane stores only extracted metrics for real-time reporting. This segmentation allows the system to maintain data completeness for analysis while reducing retrieval time for real-time operations by querying only the optimized second plane.
Solution Approach 2:
The patent performs preliminary extraction and storage of metrics during the log intake process itself, rather than extracting them on-demand during queries. This preliminary action pre-processes the data, ensuring that when real-time reporting is needed, the required metrics are already available in the second data plane, significantly reducing retrieval time.
3Adaptability or versatility
If the system queries distributed log shards for real-time dashboard analytics, then the system can provide comprehensive analytics coverage, but the query execution time increases significantly
Solution Approach 1:
The patent extracts only the necessary metrics from the distributed log shards and stores them in a centralized second data plane that is optimized for real-time querying. This extraction eliminates the need to scan through all distributed shards during real-time reporting, dramatically reducing query execution time while maintaining comprehensive analytics coverage through the extracted metrics.
Solution Approach 2:
The patent introduces a second data plane as an intermediary between the distributed log storage and the real-time reporting services. This intermediary layer receives and stores extracted metrics from the distributed shards, allowing real-time queries to be answered by querying the intermediary rather than the distributed shards directly, thus reducing query execution time while maintaining analytics coverage.
Data Source
AI summary
This disclosure describes how data supporting real-time reporting services can be cached during a log intake process. In particular, instead of caching all the log data being generated by an operational system, only the log data relevant to existing queries associated with the real-time reporting services are cached. In some embodiments, only particular metrics contained within the log data are stored for rapid access by the real-time reporting services.