LSM Tree Time Series Metrics Storage Sequential Writes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High volume time series metrics data processing in cloud computing environments faces scalability issues due to random disk I/O operations, leading to poor storage performance and unsustainability, especially in large cloud computing data centers.
Innovation Solution
Implementing a Log-Structured-Merge (LSM) tree data structure for storing time series metrics data, which sequentializes writes by co-locating data by series, and using separate buckets for different resolutions to avoid data churn during aggregation, along with sharding and integrated indexing for efficient query processing and standalone operation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a typical write strategy is used for storing time series metrics data points, then data can be stored in the database, but random disk I/O operations occur causing storage to scale poorly and become unsustainable
Solution Approach 1:
The patent segments the time series metrics data by organizing it into multiple time series families, where each family is stored in a separate file. This segmentation transforms the random I/O pattern into sequential I/O within each file while maintaining data integrity and storage reliability.
Solution Approach 2:
The patent introduces a new organizational dimension by sorting data within files not just by time but also by metric name and host name. This multi-dimensional organization enables efficient sequential writes while maintaining fast query performance across different access patterns.
2Quantity of substance
If the number of time series metrics data points per unit time grows very large, then more data can be tracked, but the random nature of I/O operations causes storage to scale poorly
Solution Approach 1:
The patent performs preliminary sorting of data by time, metric name, and host name before writing to disk. This preliminary organization ensures that even as data volume grows, the I/O operations remain sequential and predictable, preventing complexity from increasing with data quantity.
Solution Approach 2:
The patent changes the I/O operation parameter from random access to sequential access by reorganizing the data structure. This parameter change allows the system to handle millions of data points per minute with linear scalability rather than exponential complexity growth.
3Measurement precision
If data is stored with high resolution, then query precision is improved, but data churn during aggregation increases
Solution Approach 1:
The patent applies local quality by storing data at its original high resolution in the source file, then creating aggregated views only where needed. This allows query precision to be maintained at the source while minimizing data churn by performing aggregation selectively rather than universally.
Solution Approach 2:
The patent creates aggregated views as copies or projections of the original high-resolution data rather than continuously transforming it. This copying approach maintains measurement precision for queries while reducing data churn by avoiding repeated aggregation operations on the same data.
Data Source
AI summary
Embodiments of the present invention include a method of processing time series metrics (TSM) data. The method includes receiving a TSM data point for a metric, the TSM data point indicating an identifier (ID) of the metric, getting a persistence policy associated with the metric, selecting a first resolution database according to the persistence policy, storing the TSM data point in the first resolution database, selecting a second resolution database according to the persistence policy; and when the second resolution database is defined, storing the TSM data point in a staging file for the second resolution database.