Distributed Query Execution for Time-Windowed Statistical Functions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in efficiently computing statistical functions over large, time-based datasets stored in heterogeneous and unsorted orders, leading to slow performance and difficulty in managing long-running computations.
Innovation Solution
A distributed query execution system that splits statistical function computations into accumulator and aggregator tasks, executed in parallel across a fleet of worker nodes, allowing for scalable and manageable processing of large datasets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If statistical functions are computed over large time-based datasets using traditional sequential processing, then computation accuracy is maintained, but computation time becomes extremely long and system scalability is limited
Solution Approach 1:
The patent divides the computation of statistical functions into two distinct task types: accumulator tasks that process individual data records and generate intermediate results, and aggregator tasks that combine intermediate results to produce final statistical functions. This segmentation enables parallel execution across multiple worker nodes, dramatically improving computation speed while maintaining manageable system complexity through clear task separation.
2Quantity of substance
If statistical computations are performed over very large datasets with many time windows, then comprehensive analysis coverage is achieved, but computation time increases to hours and becomes difficult to manage
Solution Approach 1:
The system performs preliminary accumulation of intermediate results by accumulator tasks before final aggregation. Each accumulator task processes data records and generates intermediate statistical results that are stored and then combined by aggregator tasks. This preliminary action enables the system to handle very large datasets with many time windows efficiently, as the accumulation phase can be parallelized across multiple worker nodes, reducing overall computation time from hours to manageable levels.
3Adaptability or versatility
If log data is stored in heterogeneous ways across multiple storage locations without time-sequenced ordering, then data storage flexibility is improved, but tracking and managing computation progress becomes significantly more difficult
Solution Approach 1:
The patent implements a job manager that tracks the execution status of accumulator and aggregator tasks, providing feedback on computation progress. The system monitors which accumulator tasks have completed and which aggregator tasks are ready to execute, enabling effective management of computation workflows even when data is stored heterogeneously across multiple locations. This feedback mechanism allows the system to adapt to different storage configurations while maintaining ease of computation management.
Data Source
AI summary
Systems and methods are disclosed to implement a distributed query execution system that performs statistical operations on specified time windows over time-based datasets. In embodiments, the query system splits a statistical function into a set of parallel accumulator tasks that correspond to different portions of the dataset and/or function time windows. The accumulator tasks are executed in parallel by individual accumulator nodes to generate individual statistical result structures. The structures are then combined by an aggregator node to produce an aggregate result structure that indicates the results of the statistical function over the time windows. In embodiments, the accumulator and aggregator tasks are implemented and executed using a programmable task execution framework that allows developers to define custom accumulator and aggregator tasks. Advantageously, the query system allows queries with time-windowed statistical functions to be parallelized across a group of worker nodes and scaled to very large datasets.