Distributed Query Execution with Two-Tiered Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing relational database management systems (RDBMSs) face inefficiencies when handling queries involving large data sets, such as those with trillions of records and petabytes of data, often requiring massive server clusters and struggling with sophisticated search techniques like JOIN operations or regular expression searches, even with proper indexing.
Innovation Solution
A system that indexes files using a two-tiered index comprising an index dictionary and a Bloom filter, dynamically generates optimized scripts for querying, and distributes these scripts across multiple computing devices for parallel execution, allowing for efficient time-based file searching and aggregation of results.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If existing RDBMS solutions are used to query massive data sets (trillions of records, petabytes of data), then proper indexing can avoid iterating through every record, but the system breaks down when attempting queries involving such massive data sets and cannot effectively perform sophisticated commands like JOIN operations or regular expression searches
Solution Approach 1:
The patent segments the massive data set into multiple partitions distributed across a cluster of computing devices. Each partition is independently queryable, allowing the system to handle trillions of records by dividing the workload. The segmentation enables parallel processing of sophisticated queries like JOIN operations and regular expression searches across different partitions simultaneously.
Solution Approach 2:
The patent introduces a query planning system as an intermediary that translates high-level query specifications into distributed execution plans. This intermediary layer coordinates the sophisticated search operations across multiple partitions and computing devices, enabling complex queries to be executed efficiently on massive data sets without requiring the entire data set to be loaded into memory.
2Productivity
If massive clusters of servers are deployed to query petabytes of data, then more computing power is available, but the hardware investment becomes proportional to the data size and complexity
Solution Approach 1:
The patent creates a universal query execution engine that can handle multiple types of queries (SELECT, JOIN, regular expression searches, aggregations) on diverse data formats (Parquet, JSON, CSV) using the same infrastructure. This multi-functional system eliminates the need for specialized hardware for each query type, allowing a single cluster to efficiently process various sophisticated queries on petabytes of data without proportional hardware increases.
Solution Approach 2:
The patent implements dynamic query planning and execution that adapts to the specific characteristics of each query and data distribution. The system dynamically determines the optimal execution strategy, partition selection, and resource allocation based on query complexity and data characteristics, maximizing the utilization of available hardware resources rather than requiring excess capacity for all possible query scenarios.
3Speed
If traditional indexing methods are used on massive data sets, then query speed can be improved for simple queries, but the system fails to allow sophisticated search techniques like search by regular expression and JOIN operations
Solution Approach 1:
The patent adds a new dimension to traditional indexing by implementing a hierarchical indexing structure that works across distributed partitions. Instead of a single flat index, the system creates indexes at multiple levels (partition level, file level, and record level) that enable sophisticated queries to be pruned and optimized at each level, maintaining high speed for both simple and complex queries including regular expression searches and JOIN operations.
Data Source
AI summary
Computer-implemented methods and systems are disclosed for receiving and indexing a plurality of files for later querying, for dynamically generating scripts to be executed during a query of a data store, and for horizontally distributing a query and aggregating results of the distributed query.