Cardinality-Based Index Caching for Time Series Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional techniques, such as relational databases, struggle to manage large quantities of time-sensitive data due to scalability limitations and increased latency in query processing, especially for high cardinality queries.
Innovation Solution
A cardinality-based multi-tier index caching approach is implemented, where high cardinality indices are cached in a local memory cache and low cardinality indices are cached in a distributed memory cache, optimizing query processing latency and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional relational databases are used to manage time series data, then data management is simple and scalable, but query processing latency increases and scalability is limited for large quantities of time-sensitive data
Solution Approach 1:
The patent segments the index caching system into two distinct parts: a local memory cache for high cardinality indices and a distributed memory cache for low cardinality indices. This segmentation allows each cache type to be optimized for its specific use case, with local cache providing fast access for frequent high-cardinality queries and distributed cache providing cost-effective storage for lower-priority indices, thereby resolving the latency issue without sacrificing scalability
Solution Approach 2:
The patent introduces an intermediary mechanism (the hybrid caching system with cardinality-based routing) that sits between the query processing layer and the underlying data storage. This intermediary intelligently routes index access requests based on cardinality thresholds, providing fast local access when needed while maintaining the ability to access distributed storage, thus improving query processing speed without losing the scalability benefits of distributed systems
2Loss of time
If high cardinality indices are cached in distributed memory cache, then cost efficiency is maintained, but query processing latency increases for high cardinality queries
Solution Approach 1:
The patent applies local quality by making different parts of the caching system have different characteristics: local memory cache is used for high cardinality indices that require fast access, while distributed memory cache is used for low cardinality indices where cost efficiency is more important. This differentiation resolves the contradiction by providing fast local access for time-critical operations while maintaining cost efficiency for less time-sensitive operations, without requiring a completely complex unified system
3Reliability
If all indices are cached in local memory, then query processing latency is reduced, but cost efficiency decreases and scalability is limited
Solution Approach 1:
The patent implements a dynamic caching strategy where the system adaptively routes index access based on cardinality characteristics rather than using a static all-or-nothing approach. High cardinality indices are dynamically placed in local memory for fast access, while low cardinality indices are placed in distributed memory for cost efficiency. This dynamic allocation resolves the contradiction by optimizing for both performance and resource consumption based on the specific characteristics of each index
Data Source
AI summary
In a computer-implemented method for cardinality-based index caching of time series data, a cardinality of an index of a time series data monitoring system is determined. The cardinality of the index is compared to a cardinality threshold. Responsive to determining that the cardinality of the index exceeds the cardinality threshold, the index is cached in a local memory cache of a query node of the times series data monitoring system. Responsive to determining that the cardinality of the index does not exceed the cardinality threshold, the index is cached in a distributed memory cache of the times series data monitoring system.