Time-Series Database Partition Splitting Across Clusters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed time-series databases face challenges in managing and scaling resources efficiently, particularly in provisioning, administering, and managing large datasets, which leads to increased costs and complexity due to the scale and size of data sets.
Innovation Solution
Implementing dynamic lease assignments, heat balancing, and partition management across storage nodes and clusters, allowing for efficient distribution and management of time-series data using tile leases, random assignment, and refinement mechanisms to optimize resource utilization and minimize data movement.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored in a single database cluster, then data locality and query performance are improved, but the system lacks scalability and becomes a single point of failure
Solution Approach 1:
The patent divides the database system into multiple independent clusters, each storing a portion of the data. Partitions are distributed across clusters using hash-based routing, allowing the system to scale horizontally while maintaining data locality. Each cluster operates semi-independently, improving reliability through fault isolation.
Solution Approach 2:
The patent introduces a partition management layer that acts as an intermediary between clients and distributed clusters. This layer handles partition routing, load balancing, and failure recovery transparently, reducing the complexity burden on individual clusters while maintaining system-wide reliability.
2Productivity
If partitions are distributed across multiple clusters, then system scalability and fault tolerance are improved, but data movement complexity and network bandwidth usage increase
Solution Approach 1:
The patent pre-computes and caches partition routing information before data access operations occur. Partition metadata is stored locally at each cluster, enabling clients to determine the destination cluster without requiring real-time network queries. This preliminary organization minimizes runtime network traffic and data movement.
Solution Approach 2:
The patent uses hash functions with configurable parameters to distribute partitions across clusters. By adjusting the hash function parameters and partitioning strategy, the system can optimize the distribution pattern to minimize cross-cluster data movement while maintaining scalability and load balance.
3Reliability
If data is replicated across clusters for fault tolerance, then system reliability is improved, but storage costs and data consistency complexity increase
Solution Approach 1:
The patent implements selective replication where critical partitions are replicated across multiple clusters while less critical data maintains single-copy storage. The replication factor and target clusters are determined based on data importance and access patterns, optimizing the balance between fault tolerance and storage efficiency.
Solution Approach 2:
The patent applies partial replication rather than full replication across all clusters. Only necessary partitions are replicated to specific target clusters based on redundancy requirements, avoiding the overhead of complete data duplication while maintaining adequate fault tolerance for critical data.
Data Source
AI summary
Methods, systems, and computer-readable media for splitting partitions across database clusters in a time-series database are disclosed. A time-series database determines that a heat metric for the first tile has exceeded a threshold. The first tile represents spatial boundaries and temporal boundaries of time-series data, and a lease for the first tile is assigned to a storage node. Based (at least in part on) the heat metric, a temporal split of the first tile is performed to generate an intermediate tile representing the spatial boundaries and a later portion of the temporal boundaries. A spatial split of the intermediate tile is performed to generate second and third tiles representing two portions of the spatial boundaries and the later portion of the temporal boundaries. The storage node stores elements of the time-series data within these new boundaries to the second and third tiles.