Approximate Aggregation Queries Using Importance Filter Values

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional approximate aggregation techniques in database queries, such as random sampling, can result in excessive errors for datasets with 'heavy-tailed' distributions, leading to increased execution time and reduced accuracy due to the impact of extreme values.

Innovation Solution

Pre-populating data records with filter values indicating their importance for specific aggregation queries, allowing the query engine to count records within an importance threshold and calculate approximate aggregations, thereby reducing errors and processing time.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If random sampling is used for approximate aggregation, then processing time is reduced, but measurement precision deteriorates due to excessive errors in heavy-tailed distributions

Engineering Contradiction:

Improveprocessing timeVSAvoidaggregation accuracy

Core Design Contradiction:

ProductivityVSMeasurement precision

Solution Approach 1:

The system pre-calculates and stores filter values for each data record before the aggregation query is executed. These filter values are computed based on the aggregation function and stored in the database, so that during query execution, the system can quickly filter and sample records without performing complex calculations, thus reducing processing time while maintaining accuracy even for heavy-tailed distributions

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system transforms the original data by computing filter values that modify the sampling probability parameters. Records with extreme values in heavy-tailed distributions receive appropriate filter values that ensure they are included in the sample with sufficient probability, thereby changing the sampling parameters to match the data distribution characteristics and improve measurement precision

Inventive Principle:

Principle #35Parameter changes

2Speed

If conventional random sampling is applied to heavy-tailed data, then processing speed increases, but reliability decreases due to excessive error from missed extreme values

Engineering Contradiction:

Improvequery execution speedVSAvoiderror rate

Core Design Contradiction:

SpeedVSReliability

Solution Approach 1:

Filter values are pre-computed and stored for each record, enabling the query engine to rapidly filter and sample records during execution without performing complex calculations on the fly. This preliminary preparation maintains high query execution speed while ensuring reliable inclusion of extreme values through appropriate filter value assignment

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The filter value acts as an intermediary parameter that mediates between the raw data values and the sampling decision. By introducing this intermediate representation, the system can efficiently determine which records to include in the sample while accounting for heavy-tailed distribution characteristics, thus reducing error rates without sacrificing execution speed

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS11449514B2Approximate aggregation queries

Publication Date: 2022.09.20 INTERSET SOFTWARE

AI summary

A computing device includes a processor and a medium storing instructions. The instructions are executable by the processor to: receive a database query for an approximate aggregation of a numerical value of a plurality of records, wherein each record includes the numerical value and a filter value; in response to the database query, determine a count of records that have filter values within an importance threshold associated with the database query; and determine the approximate aggregation of the numerical value based on the count of records and the importance threshold associated with the database query.