Horizontal Time Series Data Sorting via Leader Host Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed network systems, sorting data received out of order across a horizontally scaled ingestion fleet of physical devices is challenging, particularly when data needs to be sorted without relying on a vertically scaled host for collocation.
Innovation Solution
A leader physical host generates and compiles data histograms from various physical hosts, dividing them into segments weighted by observation payload size and assigning these segments to individual hosts based on their data contribution, allowing for horizontal partitioning and sorting of time-partitioned data without the need for a vertically scaled host.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is received out of order across a horizontally scaled ingestion fleet, then data can be collected from multiple sources, but sorting and merging data becomes difficult
Solution Approach 1:
The patent segments the time series data into distinct time intervals and partitions data across multiple hosts based on these time intervals. Each host maintains data for specific time ranges, enabling parallel collection while simplifying sorting through pre-defined temporal segments rather than requiring complex global sorting operations.
Solution Approach 2:
The patent introduces a coordinator host that acts as an intermediary to manage data distribution. The coordinator receives data from multiple sources, determines ownership based on time intervals, and directs data to appropriate hosts, thereby simplifying the sorting and merging process without requiring complex peer-to-peer coordination.
2Productivity
If a vertically scaled physical host is used for collocation, then data can be sorted efficiently, but system scalability is limited
Solution Approach 1:
The patent divides the sorting function across multiple horizontally scaled hosts rather than concentrating it on a single vertically scaled host. Data is partitioned by time intervals and distributed to different hosts, maintaining sorting efficiency through parallel processing while enabling system scalability through horizontal expansion.
Solution Approach 2:
The patent transitions from vertical scaling (adding more resources to a single host) to horizontal scaling (adding more hosts to the system). By organizing data across multiple hosts in a horizontal dimension with clear ownership boundaries based on time intervals, the system achieves both scalability and efficient sorting without requiring a single powerful vertical host.
3Adaptability or versatility
If data is partitioned across multiple hosts, then system scalability is improved, but data exchange among hosts increases
Solution Approach 1:
The patent performs preliminary actions by pre-dividing time intervals and pre-determining data ownership before data arrives. The coordinator host assigns time intervals to specific hosts in advance, so data is directed to the correct host without requiring extensive exchange or reshuffling operations, thereby reducing data exchange overhead while maintaining scalability.
Solution Approach 2:
The patent implements local quality by having each host manage only the data relevant to its assigned time interval. This localized responsibility minimizes the need for hosts to exchange data with others, as each host processes only its designated portion, reducing overall data exchange overhead while enabling system scalability through horizontal partitioning.
Data Source
AI summary
A leader host obtains individual distributions of data sets ingested by individual hosts of a fleet of hosts over a domain. The leader host compiles the individual distributions over the domain to generate a compiled distribution. The leader host then partitions the domain based at least in part on the generated compiled distribution. These partitions of the partitioned domain are distributed to individual hosts of the fleet of hosts, which causes the individual hosts to process a portion of the distributed date set according to their respective partitions.