Temporal Graph Partitioning for Distributed Telemetry Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current network data analytics systems face challenges in efficiently storing and retrieving large volumes of unbounded telemetry information due to the lack of effective partitioning of temporal graphs, leading to cumbersome data management processes.
Innovation Solution
The method involves partitioning temporal graphs by creating storage blocks based on predefined label groups and sharding them into multiple shards, with each block configured to store data within a specific time range and recreated upon rollover, optimizing data distribution and retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If temporal graphs are used to store unbounded telemetry information streams, then the storage capacity is improved, but the data retrieval efficiency deteriorates due to lack of proper partitioning
Solution Approach 1:
The temporal graph data is segmented into multiple time-range partitions, where each partition stores telemetry data for a specific time window. This segmentation allows the system to handle unbounded data streams by dividing them into manageable chunks, improving both storage organization and retrieval efficiency by directing queries to specific time-range partitions rather than scanning the entire dataset.
2Loss of information
If temporal graphs store all telemetry information without partitioning, then data completeness is improved, but data management complexity increases
Solution Approach 1:
The system segments the temporal graph into multiple partitions based on time ranges, with each partition maintaining a complete subset of telemetry data for its designated time window. This segmentation reduces management complexity by allowing independent handling, storage, and retrieval of each partition while collectively preserving complete telemetry information across all partitions.
Solution Approach 2:
The system introduces a time-range dimension for partitioning, organizing telemetry data not only by graph structure but also by temporal boundaries. This additional dimensional organization allows the system to maintain data completeness while simplifying management operations through structured, time-based partitioning that enables targeted data access and reduced overhead.
3Productivity
If storage blocks are spatially and temporally aligned, then writing and indexing speeds are improved, but system complexity increases
Solution Approach 1:
Storage blocks are segmented into fixed-size partitions aligned to specific time ranges, creating a regular, predictable structure that optimizes writing and indexing operations. Each partition is independently managed with consistent formatting, enabling efficient batch operations and reduced contention while the alignment to time boundaries simplifies query routing and data lifecycle management.
Data Source
AI summary
In one embodiment, present disclosure discloses a method for partitioning a temporal graph is described. Embodiments of the method comprises creating a plurality of storage blocks for each type of the different types of graph elements based on predefined label groups, each of the plurality of storage blocks configured to store the telemetry information generated in a corresponding predefined time-range, recreating each of the plurality of storage blocks upon expiry of a configurable rollover time, and sharding each of the plurality of storage blocks into a plurality of shards based on a configurable sharding count.