Pre-Aggregated Data Tables for Adaptive Query Retrieval

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing data retrieval systems face slow query processing times, manual generation of pre-aggregated tables leads to inefficiencies, and query patterns change over time, resulting in redundant tables and inconsistent data across different data sources.

Innovation Solution

An automated system that uses offline pre-computation and online optimization components to generate and manage pre-aggregated data tables based on query patterns, employing AI models to determine query dimensions and optimize storage and retrieval processes.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Speed

If manual generation of pre-aggregated tables is performed, then query response time is improved, but productivity is worsened due to manual efforts and time consumption

Engineering Contradiction:

Improvequery response timeVSAvoidtable generation efficiency

Core Design Contradiction:

SpeedVSProductivity

Solution Approach 1:

The system performs self-service by automatically analyzing query logs and generating pre-aggregated tables without manual intervention. The query log analysis system identifies query patterns and triggers automated table generation, eliminating the need for data engineers to manually create tables while maintaining fast query responses.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The system performs preliminary action by pre-computing and storing aggregated data in advance based on predicted query patterns. The offline pre-computation component generates pre-aggregated tables before actual queries are executed, enabling fast query responses without manual table generation for each query.

Inventive Principle:

Principle #10Preliminary action

2Speed

If multiple pre-aggregated tables are created to handle different query patterns, then query performance is improved, but device complexity is worsened due to redundant tables and overlapping data

Engineering Contradiction:

Improvequery performanceVSAvoiddata table structure

Core Design Contradiction:

SpeedVSDevice complexity

Solution Approach 1:

The system creates universal pre-aggregated tables that can handle multiple query patterns through a single table structure. By using a unified schema that supports various query types and dimensions, the system eliminates the need for separate specialized tables for each query pattern, reducing redundancy while maintaining performance.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The system merges multiple potentially redundant pre-aggregated tables into a single comprehensive table structure. The offline pre-computation component identifies overlapping data and combines it into unified tables, reducing the total number of tables while preserving all necessary query optimization capabilities.

Inventive Principle:

Principle #5Merging (Combining)

3Speed

If pre-aggregated tables are created for current query patterns, then query response time is improved, but adaptability is worsened because tables do not automatically adapt to changing query patterns

Engineering Contradiction:

Improvequery response timeVSAvoidquery pattern adaptation

Core Design Contradiction:

SpeedVSAdaptability or versatility

Solution Approach 1:

The system implements feedback by continuously monitoring query logs and using this information to dynamically adjust pre-computation strategies. The query log analysis system identifies changing query patterns and triggers updates to pre-aggregated tables, ensuring the system adapts to evolving query requirements while maintaining fast response times.

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The system transitions from static pre-aggregated tables to dynamic structures that can evolve with changing query patterns. The offline pre-computation component periodically regenerates tables based on current query log analysis, allowing the table schema and aggregation strategies to adapt automatically to new query requirements.

Inventive Principle:

Principle #15Dynamics

4Reliability

If data is aggregated from multiple sources to ensure consistency, then data reliability is improved, but loss of time is worsened due to additional processing and coordination requirements

Engineering Contradiction:

Improvedata consistencyVSAvoiddata processing time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system performs preliminary action by pre-computing and pre-validating data consistency across multiple sources before actual queries are executed. The offline pre-computation component aggregates and reconciles data from multiple sources in advance, ensuring consistency is established beforehand rather than during query execution, thus reducing processing time while maintaining reliability.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS20250348489A1Automatic query and data retrieval optimization through procedural generation of data tables from query patterns

Publication Date: 2025.11.13 PAYPAL INC

AI summary

Latency, response times, and efficiency improvements for data querying are provided herein, particularly in the context of querying large database systems and data tables from disparate data sources. There are provided systems and methods for automatic query and data retrieval optimization through procedural generation of data tables from query patterns. A service provider may utilize different computing services for query processing and data retrieval for different applications and services used by internal and/or external users. Instead of querying large database systems and numerous data tables, pre-aggregated data tables may instead be used and searched by procedurally generating such tables based on precomputation rules and query patterns. Once patterns have been identified in queries, corresponding data may be aggregated from data sources in a pre-aggregated data table. Query optimization rules may then be used to have these data tables queried in place of their original sources.