Ingestion Router Clustering for Distributed Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing algorithms for clustering time series data are less efficient in distributed storage environments, requiring significant data shuffling across networks, which hampers performance in scalable storage systems.
Innovation Solution
A clustering algorithm that groups data based on a hierarchy of attribute values, using partition tables to spatially organize data at the point of ingestion, minimizing the need for data movement and optimizing storage and querying efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If existing clustering algorithms are used in distributed storage environments, then data can be clustered, but significant data shuffling across the network is required, reducing efficiency
Solution Approach 1:
The patent applies preliminary action by performing data clustering at the point of ingestion, before data is stored in the distributed database. The ingestion router clusters incoming data elements based on attribute values and stores them directly in appropriate data stores, eliminating the need for subsequent data shuffling operations that would be required if clustering were performed after data ingestion.
Solution Approach 2:
The patent introduces an ingestion router as an intermediary component between data producers and the distributed database. This router performs the clustering function by examining attribute values of incoming data elements, determining appropriate data stores, and routing data accordingly, thereby eliminating the need for data shuffling that would occur with traditional post-ingestion clustering approaches.
2Adaptability or versatility
If data is stored in a single location for clustering, then clustering algorithms can be applied, but the system lacks scalability for large distributed data environments
Solution Approach 1:
The patent applies local quality by enabling each data store in the distributed system to maintain its own local clustering structure based on attribute values. Each data store independently organizes the data elements it receives, allowing the system to scale across multiple locations while maintaining efficient local clustering without requiring centralized data aggregation.
Solution Approach 2:
The patent segments the clustering function across multiple independent data stores in the distributed system. Rather than requiring all data to be in a single location, the system divides data storage and clustering into separate segments, with each data store handling its own clustering locally. This segmentation enables scalability while maintaining clustering efficiency.
3Speed
If data is not clustered at ingestion, then storage is simpler, but query performance decreases due to larger data filtering requirements
Solution Approach 1:
The patent applies preliminary action by performing data clustering at the point of ingestion, before data is stored in the distributed database. The ingestion router clusters incoming data elements based on attribute values and stores them directly in appropriate data stores, eliminating the need for subsequent data shuffling operations that would be required if clustering were performed after data ingestion.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Techniques are described for clustering data at the point of ingestion for storage using scalable storage resources. The clustering techniques described herein are used to cluster time series data in a manner such that data that is likely to be queried together is localized to a same partition, or to a minimal set of partitions if the data set is large, where the partitions are mapped to physical storage resources where the data is to be stored for subsequent processing. Among other benefits, the clustered storage of the data at the physical storage resources can reduce an amount of data that needs to be filtered by many types of queries, thereby improving the performance of any applications or processes that rely on querying the data.