Time-Series Sketches for Real-Time Approximate Query Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing query processing technologies struggle to provide real-time and accurate results, especially for large datasets, and existing summarization approaches either lack real-time responses, provide inaccurate results, or have memory inefficiencies when dealing with univariate and multivariate queries involving numerical and categorical data.
Innovation Solution
Generating a time series of sketches for each column of tabular data, where numerical data is divided into bins and categorical data is processed to identify top-k values, allowing for efficient storage and retrieval of sketches to answer queries with high accuracy and manageable memory footprint.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If existing summarization approaches are used for large datasets, then query processing can be performed, but real-time responses cannot be provided and accuracy deteriorates
Solution Approach 1:
The patent segments the data processing approach by creating separate sketch data structures for different columns and data types (numerical vs. categorical). Each column is processed independently to generate its own time series of sketches, allowing parallel processing and faster query response while maintaining accuracy through specialized handling of each data type's characteristics
Solution Approach 2:
The patent performs preliminary action by pre-computing and storing time series of sketches for each column before queries are executed. These sketches capture statistical properties and patterns over time, so when a query arrives, the system can quickly retrieve and combine relevant sketches instead of processing raw data, enabling real-time responses without sacrificing accuracy
2Measurement precision
If detailed data summarization is performed to maintain accuracy, then query result precision improves, but memory consumption increases
Solution Approach 1:
The patent applies local quality by creating customized sketch data structures tailored to specific data types and column characteristics. Numerical columns receive different sketch treatments than categorical columns, and each sketch captures only the local statistical properties relevant to that column's query patterns, reducing overall memory usage while maintaining query accuracy
Solution Approach 2:
The patent uses partial action by generating sketches that capture sufficient statistical properties for accurate query processing without storing complete detailed data. The sketches contain condensed representations (e.g., mean, variance, frequency distributions) that are adequate for most query operations, avoiding the memory overhead of storing full data while maintaining acceptable accuracy
3Adaptability or versatility
If time series data is processed to provide historical query capabilities, then query versatility improves, but processing complexity increases
Solution Approach 1:
The patent implements universality by designing a unified sketch data structure and processing framework that handles both numerical and categorical data types, supports multiple query types (univariate and multivariate), and works across different time periods. This universal approach reduces processing complexity compared to maintaining separate systems for each data type and query scenario
Data Source
AI summary
Some aspects relate to technologies for generating time series of sketches and using the time series of sketches for approximate query processing. In accordance with some aspects, tabular data is accessed that has a number of columns. Responsive to identifying a first column as comprising numerical data, sketches are generated for the numerical data for each of a number of time steps, and the sketches for the numerical data are stored as a first time series of sketches. Responsive to identifying a second column as comprising categorical data, sketches are generated for the categorical data for each of the time steps, and the sketches for the categorical data are stored as a second time series of sketches. When a query is received, a response to the query is provided using sketches from the first time series of sketches and/or the second time series of sketches.