Temporal Splits for Time-Series Database Auto-Scaling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The complexity of managing and scaling large distributed time-series databases leads to inefficiencies in resource utilization and data durability due to the need for frequent spatial splits, which require significant data movement and can result in data accumulation and throttling.
Innovation Solution
Implementing temporal splits instead of spatial splits for two-dimensional tiles in a time-series database, reducing data movement and allowing for faster scaling and improved durability by partitioning data along temporal boundaries without requiring data redistribution across storage nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Volume of stationary object
If spatial splits are performed to scale the database, then storage capacity is increased, but data movement and system complexity increase significantly
Solution Approach 1:
The patent segments the time-series database into multiple independent time-range partitions, where each partition stores data for a specific time period. This segmentation allows the database to scale by adding new time-based partitions rather than redistributing existing data spatially, thereby increasing storage capacity without proportionally increasing system complexity.
Solution Approach 2:
The patent introduces a temporal dimension for data partitioning, moving away from traditional spatial-only partitioning. By organizing data along the time axis into sequential time-range partitions, the system can scale storage capacity by simply adding new time-based partitions without requiring complex spatial redistribution of data across nodes.
2Volume of stationary object
If spatial splits are performed to increase storage capacity, then more data can be stored, but data movement and throttling occur
Solution Approach 1:
The database is segmented into time-range partitions that are naturally ordered by time. When storage capacity needs to be increased, new partitions are created for future time ranges rather than performing spatial splits that require moving existing data. This eliminates data movement overhead while still increasing storage capacity.
Solution Approach 2:
The system proactively creates new time-range partitions before data needs to be stored, rather than performing reactive spatial splits when storage is full. This preliminary action of pre-allocating time-based storage space eliminates the need for data movement during scaling operations.
3Volume of stationary object
If frequent spatial splits are performed, then storage capacity is maintained, but query performance and availability deteriorate
Solution Approach 1:
The database is segmented into time-range partitions that are optimized for sequential access patterns. Queries can be efficiently routed to specific time-range partitions based on time predicates, improving query performance. This segmentation approach avoids the performance deterioration associated with frequent spatial splits while maintaining storage capacity.
4Volume of stationary object
If spatial splits are performed, then storage is scaled, but data accumulation and throttling increase
Solution Approach 1:
The database is segmented into time-range partitions with clear temporal boundaries. Each partition can be independently managed, replicated, and recovered. This segmentation improves data durability by isolating failures to specific time ranges and enabling targeted recovery operations, while avoiding the data accumulation and throttling issues associated with spatial splits.
Data Source
AI summary
Methods, systems, and computer-readable media for auto-scaling using temporal splits in a time-series database are disclosed. One or more stream processors of a time-series database write time-series data of a plurality of time series into a plurality of two-dimensional tiles, including an individual tile representing spatial boundaries and temporal boundaries. The heat of time-series data written to the individual tile exceeds a threshold. The current time is compared to the temporal boundaries, and the comparison indicates that the current time is beyond a threshold point within the temporal boundaries. Based at least in part on the comparison, a split is performed of the individual tile into a first new tile and a second new tile. The first new tile represents the spatial boundaries and a first portion of the temporal boundaries, and the second new tile represents the spatial boundaries and a second portion of the temporal boundaries.