Partition Segmenting in Distributed Time-Series Database
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed time-series databases face challenges in managing out-of-order data due to limited storage capacity in closed tiles, leading to data discard and inefficiencies in data management and querying.
Innovation Solution
The implementation of a chain of segments with shared temporal and spatial boundaries, allowing for the extension of storage capacity by adding new segments on different hosts, enabling the storage of out-of-order data and efficient deduplication, while maintaining data boundaries and ensuring high availability and low latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If storage capacity in closed tiles is limited to maintain data boundaries, then data organization and query efficiency are improved, but out-of-order data cannot be stored and must be discarded
Solution Approach 1:
The patent divides a closed tile into multiple segments (first segment, second segment, third segment) that can independently store data. This segmentation allows the tile to maintain its temporal and spatial boundaries while expanding storage capacity through multiple segments, enabling out-of-order data to be stored in subsequent segments without violating data boundary integrity.
Solution Approach 2:
The patent introduces a new dimension to storage by adding segments beyond the traditional single-tile structure. Instead of expanding storage within the same tile boundary, the system creates additional segments (first segment, second segment, third segment) that extend the storage capacity in a dimensional sense, allowing out-of-order data to be stored while maintaining original tile boundaries.
2Adaptability or versatility
If storage capacity is increased by adding segments to tiles, then out-of-order data can be stored, but system complexity increases
Solution Approach 1:
The patent introduces a control plane as an intermediary that automatically manages segment creation, data distribution, and deduplication across multiple segments. This intermediary abstracts the complexity of managing segmented tiles from the coordination layer, which only needs to interact with the control plane, thereby reducing system complexity despite increased storage capacity.
Solution Approach 2:
The system implements automatic deduplication and segment management where the control plane autonomously handles data placement, segment creation, and redundancy management. This self-service mechanism reduces the need for manual intervention and simplifies the overall system architecture by automating complex segment chain management tasks.
3Reliability
If data is stored in multiple segments across different hosts, then availability and scalability are improved, but data management and querying become more complicated
Solution Approach 1:
The patent creates a universal segment interface that can be accessed through a common coordination layer, allowing the same data management operations to work across single-segment and multi-segment tiles uniformly. This multi-functionality enables the system to handle segmented data transparently, maintaining ease of operation while improving availability through distributed segment storage across different hosts.
4Quantity of substance
If deduplication is performed across multiple segments, then storage efficiency is improved, but processing time increases
Solution Approach 1:
The patent performs deduplication proactively during the data ingestion phase rather than during queries. The control plane deduplicates data as it is being stored across segments, eliminating duplicate data before it consumes storage resources. This preliminary deduplication action improves storage efficiency without impacting query performance, as the deduplication work is completed during the faster write operation.
Data Source
AI summary
Methods, systems, and computer-readable media for partition segmenting in a distributed time-series database are disclosed. A first host receives a request to store a data element in a tile comprising linked segments including a head segment and a tail segment. The head and tail segments are associated with the same spatial and temporal boundaries, and the data element is within those boundaries. The first host determines that the data element is not stored in the head segment and forwards the data element to a next segment of the tile. An additional host receives a request to store the data element in the tail segment. The additional host determines that the data element is not stored in the tail segment and then stores the data element in the tail segment.