Multi-tenant Monitoring System with Zero Setup Pivot Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing monitoring solutions in computing environments face limitations in flexibility and predictability when performing queries on aggregated monitoring data, leading to unpredictable resource usage and scalability issues as the amount of data increases.
Innovation Solution
A multi-tenant monitoring system that receives and sorts monitoring data in un-aggregated or semi-aggregated format, storing it in data files without additional indexing, allowing for flexible pivot queries along various dimensions and reducing computing resource usage by performing aggregation only at query time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If monitoring data is pre-aggregated and stored in a database structure, then query performance is improved, but query flexibility is limited based on how the data was aggregated
Solution Approach 1:
The patent applies preliminary action by pre-sorting monitoring data at ingestion time according to specific sort orders (e.g., by host, by metric, by time). This preliminary organization enables both fast queries and flexible pivoting without requiring pre-aggregation, as the sorted structure allows efficient data retrieval and re-grouping for different query dimensions.
Solution Approach 2:
The patent implements dynamics by allowing the sort order to be dynamically selected based on query requirements. The system can pivot data along different dimensions (host, metric, time, dimension) by applying appropriate sort orders, making the data structure adaptable to various query flexibility needs while maintaining query performance.
2Speed
If monitoring data is stored with detailed indexing for fast retrieval, then query speed is improved, but device complexity and storage overhead increase
Solution Approach 1:
The patent applies segmentation by dividing the indexing approach into simple time-based partitions and sort-order-based organization. Instead of complex multi-dimensional indexes, the system segments data by time periods and applies sort orders, reducing indexing complexity while maintaining fast retrieval through the organized structure.
Solution Approach 2:
The patent changes the parameter of data organization from complex indexed structures to sorted sequences with selectable sort orders. This parameter change simplifies the indexing mechanism while enabling fast data retrieval through efficient sorting algorithms and time-based partitioning.
3Quantity of substance
If pre-aggregation is performed to reduce data volume, then storage requirements are reduced, but query options are limited and additional aggregation steps are required
Solution Approach 1:
The patent extracts the aggregation function from the storage layer and moves it to the query layer. Instead of pre-aggregating data and storing reduced volumes, the system stores complete sorted monitoring data and performs aggregation only when needed for specific queries, preserving full query options while computing aggregates on-demand.
4Adaptability or versatility
If comprehensive indexing is implemented for all query dimensions, then query flexibility is improved, but computing resources and time required for maintenance increase
Solution Approach 1:
The patent applies universality by using a single sort order mechanism that serves multiple query dimensions. The same sorted data structure supports pivoting along different dimensions (host, metric, time, dimension) without requiring separate indexes for each, reducing maintenance time while maintaining query flexibility.
Data Source
AI summary
Monitoring data can be sent, received, stored, and accessed with zero setup. Customers can send monitoring data to be stored without the customers having to perform any setup ahead-of-time, such as arbitrary lists of key-value pairs. Customers can perform zero setup pivot queries using stored monitoring data. For example, users can perform pivot queries using any available stored information, such as available metric types and dimensions, without any setup needed ahead-of-time.