Parallel Query Trees for Large Time Series Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in efficiently storing and processing large, continuously updating time series datasets due to memory and processing constraints, making it difficult to perform complex queries on these datasets.
Innovation Solution
A system that associates time series datasets with nodes, resolves nodes based on user queries, and generates parallelized query trees for processing by a compute cluster, leveraging libraries for usability and efficient query generation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If time series datasets are stored in a traditional centralized manner, then data integrity is maintained, but memory and processing constraints prevent efficient querying of large datasets
Solution Approach 1:
The patent segments the centralized time series dataset into multiple distributed partitions stored across different nodes in a compute cluster. Each partition can be independently queried and processed, allowing the system to handle large datasets that exceed single-machine memory constraints while maintaining query efficiency through parallel processing across multiple nodes.
2Loss of time
If queries are processed sequentially on a single device, then implementation is simple, but processing time increases significantly for large datasets
Solution Approach 1:
The query processing architecture is segmented into multiple parallel workers distributed across a compute cluster. Each worker can independently execute query operations on different data partitions simultaneously, reducing overall query execution time from sequential single-device processing to parallel multi-device processing.
Solution Approach 2:
The patent transitions from single-dimensional sequential processing to multi-dimensional parallel processing by distributing query execution across multiple spatial dimensions (different compute nodes). This dimensional expansion allows concurrent query execution on multiple fronts, dramatically reducing total processing time for large datasets.
3Speed
If user devices attempt to store and process large time series datasets locally, then data access is fast, but memory constraints make it impossible for user devices
Solution Approach 1:
The patent introduces a distributed file system and compute cluster as intermediaries between user devices and the stored time series data. User devices can quickly access data through the distributed file system without storing large datasets locally, while the compute cluster provides powerful processing capabilities for analyzing large datasets that exceed user device memory constraints.
Data Source
AI summary
Systems and methods are provided for enhanced processing of time series data via parallelization of instructions. An example method includes receiving a query indicating time series datasets and operations to be performed on the time series datasets. Nodes associated with the query are identified, with each node associated with a time series dataset. Nodes associated with operations to be performed are generated. The nodes are assembled into query tree, with parent nodes of the query tree indicating operations that are to be applied to children nodes. Instructions for processing the query tree are generated. At least a subset of the instructions is provided to one or more compute systems for processing in parallel. Results are received, and presented in a user interface.