Time-Series Sketch Binning for Approximate Queries on Mixed Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing query processing technologies struggle to provide real-time and accurate responses for large datasets due to inefficiencies in data summarization, particularly when dealing with univariate and multivariate queries involving both numerical and categorical data, and they often require large memory footprints or result in inaccurate approximate responses.
Innovation Solution
Generating a time series of sketches for each column of tabular data, where numerical data is divided into bins and categorical data is processed to identify top-k values, allowing for efficient storage and retrieval of sketches that facilitate real-time query processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data summarization is performed using traditional methods, then query processing can be done, but real-time response capability is lost due to inefficiency
Solution Approach 1:
The patent segments the data into fixed-size bins and processes each bin independently to generate sketches. This segmentation allows parallel processing of multiple bins, improving both the speed of query response and the efficiency of data summarization by dividing the large dataset into manageable chunks that can be processed simultaneously
Solution Approach 2:
The patent performs preliminary data summarization by generating sketches for each bin before queries are executed. These pre-computed sketches are stored and can be quickly retrieved and combined to answer queries in real-time, eliminating the need for expensive full-data scans during query processing
2Measurement precision
If sketches are generated for all possible values, then query accuracy is improved, but memory footprint increases significantly
Solution Approach 1:
The patent changes the parameter of data representation by transforming continuous or high-cardinality data into discrete bin indices. Instead of storing sketches for all possible original values, it stores sketches indexed by bin numbers, dramatically reducing memory footprint while maintaining query accuracy through the binning structure
Solution Approach 2:
By segmenting the value space into fixed bins, the patent reduces the infinite or very large continuous domain into a finite set of discrete segments. This segmentation allows memory-efficient storage of sketches indexed by bin identifiers rather than requiring storage for every possible input value
3Adaptability or versatility
If the system processes both numerical and categorical data with different methods, then data type versatility is improved, but system complexity increases
Solution Approach 1:
The patent implements a universal binning framework that can handle both numerical and categorical data types through a common interface. The same sketch generation and query processing logic applies to both data types, reducing system complexity despite supporting multiple data types. Numerical data is binned by value ranges while categorical data is binned by frequency, but both use the same underlying sketch structure
Data Source
AI summary
Some aspects relate to technologies for generating time series of sketches and using the time series of sketches for approximate query processing. In accordance with some aspects, tabular data is accessed that has a number of columns. Responsive to identifying a first column as comprising numerical data, sketches are generated for the numerical data for each of a number of time steps, and the sketches for the numerical data are stored as a first time series of sketches. Responsive to identifying a second column as comprising categorical data, sketches are generated for the categorical data for each of the time steps, and the sketches for the categorical data are stored as a second time series of sketches. When a query is received, a response to the query is provided using sketches from the first time series of sketches and/or the second time series of sketches.