High-Cardinality Time-Series Data Bucketing for Multi-Tenant Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High-cardinality time-series data processing in multi-tenant cloud computing systems leads to long latencies, poor performance, and increased computing resource demands due to the large amounts of data that must be stored and processed.
Innovation Solution
Collect multi-dimensional data on a per-request basis, combine objects by tenant and time window to form objects of a second type, and store them in separate text files, allowing for efficient querying of the data by retrieving and combining objects of the second type to generate query results.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If high-cardinality time-series data is stored and processed in traditional multi-tenant systems, then complete data retention and querying capability are achieved, but storage requirements and processing time increase significantly
Solution Approach 1:
The patent segments high-cardinality time-series data by dividing it into multiple time windows (e.g., 1-hour, 6-hour, 12-hour windows). Each window contains a subset of the data, allowing queries to process only relevant segments rather than the entire dataset. This segmentation reduces processing time while maintaining complete data retention capability across all windows.
Solution Approach 2:
The patent introduces a time-window dimension to organize data that traditionally only had tenant and metric dimensions. By adding this temporal dimension, data is structured in a multi-dimensional hierarchy that enables efficient filtering and aggregation, reducing processing time for queries that can operate at higher time-window levels without examining individual data points.
2Reliability
If high-cardinality time-series data is stored in traditional multi-tenant systems, then all data is retained, but storage requirements increase significantly
Solution Approach 1:
The patent merges data from multiple tenants that share common characteristics within the same time window into consolidated data structures. By combining data that can be processed together, the system reduces redundant storage of metadata and indexing information while retaining all original data points for complete data retention.
Solution Approach 2:
Data is segmented into time-window buckets that can be stored and managed independently. This allows the system to retain complete data across segments while enabling selective loading and processing of only those segments relevant to current queries, effectively reducing the storage footprint accessible at any given time.
3Adaptability or versatility
If traditional data processing methods are used for high-cardinality time-series data, then comprehensive querying is possible, but computing resources are consumed excessively
Solution Approach 1:
The patent performs preliminary aggregation and organization of data into time-window structures during data ingestion, before queries are executed. This preliminary action pre-computes and stores data in an optimized format, so that subsequent queries can retrieve and process pre-organized data rather than raw data, significantly reducing computing resource consumption while maintaining comprehensive querying capability.
Solution Approach 2:
The system dynamically adjusts the level of aggregation and time-window granularity based on query requirements. For simple queries, highly aggregated data is used requiring minimal computing resources; for complex queries requiring detailed analysis, the system can drill down to finer granularities. This dynamic adaptation maintains versatile querying capability while optimizing resource usage for each specific query.
Data Source
AI summary
A method and apparatus for collecting and supporting querying of multi-dimensional data pertaining to usage of software and/or hardware to service tenant requests in a multi-tenant cloud computing system where the multi-dimensional data is initially captured on a per request basis and recorded in objects of a first type that store data pertaining to a specific request, specific tenant, specific host and specific time. The objects of the first type are combined by time windows to form objects of a second type. The objects of a second type are stored in another system as separate text files. Responsive to a query for multi-dimensional data for a specific tenant that spans an interval of multiple time windows, the objects of the second type for the specific tenant and time interval are combined across all hosts to generate a query result, and the query result is returned.