Cardinality-Based Rate Limiting for Cloud Time-Series Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing cloud computing systems are inefficient in handling high-cardinality time-series data, leading to excessive resource allocation and storage issues, as they only limit data by size rather than cardinality, resulting in unnecessary hardware allocation and resource waste.
Innovation Solution
Implementing a system with cardinality servers and a central assessment service to detect and block high-cardinality data streams, using the HyperLogLog algorithm to estimate cardinality across multiple database clusters, allowing or blocking further data from tenants based on predetermined thresholds, and informing tenants of blacklisting for correction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data storage systems allocate additional shards to handle increasing cardinality, then the system can accommodate more distinct time-series data, but hardware resources and storage costs increase excessively
Solution Approach 1:
The system performs preliminary cardinality assessment before data is fully ingested by evaluating a sample of the data stream. This early evaluation allows the system to predict total cardinality and make allocation decisions in advance, avoiding the need to provision hardware for worst-case scenarios and thus reducing overall hardware resource requirements.
Solution Approach 2:
The system employs automated cardinality estimation algorithms that self-evaluate incoming data streams without requiring manual intervention or over-provisioning. The automated assessment service continuously monitors and adjusts shard allocation based on actual cardinality measurements, allowing the system to serve itself efficiently without excessive hardware reserves.
2Reliability
If the system processes all incoming time-series data without cardinality-based rate limiting, then data completeness is maintained, but computing resources and network traffic are wasted on anomalous high-cardinality streams
Solution Approach 1:
The system implements a feedback mechanism where cardinality assessment results are used to dynamically adjust rate limiting decisions. The automated assessment service continuously monitors cardinality metrics and provides feedback to the rate limiting component, which then adjusts data ingestion rates accordingly. This closed-loop control ensures that computing resources are allocated efficiently based on actual data characteristics rather than processing all data uniformly.
Solution Approach 2:
The system changes the parameter of data ingestion rate based on cardinality assessment results. For normal data streams, the system maintains high ingestion rates to ensure data completeness. For anomalous high-cardinality streams, the system dynamically reduces the ingestion rate or applies rate limiting, thereby conserving computing resources and network bandwidth while preserving the integrity of legitimate data flows.
3Productivity
If the system adds cardinality assessment services and rate limiting mechanisms, then resource efficiency improves, but system complexity increases
Solution Approach 1:
The system segments the data processing architecture into distinct functional components: an automated cardinality assessment service that evaluates data streams, a rate limiting service that enforces constraints based on assessment results, and the core data storage system. This segmentation allows each component to perform its specialized function efficiently and enables independent optimization and maintenance of each module, managing overall system complexity through modular design.
Data Source
AI summary
A cloud-based service records time-series data of a plurality of metrics from a plurality of tenants. The time-series data comprises a series of measures of a metric at sequential points in time. The time-series data is recorded in database tables in a manner that consumes more resources when the cardinality of the time-series data is high. When the cardinality of the data for the tenant is too high, the tenant is blacklisted and further data for the tenant is not stored. In addition to identifying incoming time-series data by the tenant that produced it, the data may also identify its type. In this case, the cardinality of each type of data may be determined and the types of data may be individually allowed or blacklisted for the tenant.