Dynamic Script Generation for Distributed Query Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing relational database management systems (RDBMS) face inefficiencies when handling queries involving large datasets, such as those with trillions of records and petabytes of data, often requiring massive server clusters and struggling with sophisticated search techniques like JOIN operations, even with proper indexing.
Innovation Solution
A system that indexes files using a two-tiered index comprising an index dictionary and a Bloom filter, dynamically generates optimized scripts for querying, and distributes these scripts across multiple computing devices for parallel execution, allowing for efficient time-based file searching and aggregation of results.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If existing RDBMS solutions are used to query large datasets, then proper indexing can improve query speed, but the system breaks down when attempting to execute queries involving trillions of records comprising petabytes of data
Solution Approach 1:
The patent divides the massive dataset into multiple partitions stored across different computing devices. Each partition is independently queryable, and the system coordinates queries across partitions to handle datasets of trillions of records. This segmentation allows the system to maintain reliability and query capability at scales that would overwhelm traditional RDBMS solutions.
Solution Approach 2:
The patent introduces a new dimension of distributed computing architecture, moving from centralized RDBMS to a distributed file system across multiple computing devices. This dimensional shift enables the system to handle petabytes of data by distributing storage and processing across the network, rather than relying on a single centralized system.
2Productivity
If massive clusters of servers are deployed to query petabytes of data, then query capability improves, but hardware investment increases proportionally
Solution Approach 1:
The patent creates a universal query processing framework that can handle diverse query types (SELECT, JOIN, aggregation, regex searches) across distributed data partitions. This multi-functional system replaces the need for specialized hardware clusters, allowing a single distributed architecture to perform multiple query operations that previously required different dedicated systems.
Solution Approach 2:
The patent uses index structures (Bloom filters and index dictionaries) that are distributed across the network, allowing computing devices to copy and cache index data locally. This copying mechanism enables fast query processing without requiring every device to access the entire petabyte-scale dataset, reducing the need for proportional hardware investment.
3Productivity
If traditional indexing methods are used, then query efficiency is maintained for millions of records, but the system cannot effectively perform sophisticated searches like JOIN operations or regex on petabytes of data
Solution Approach 1:
The patent pre-computes and distributes index structures (Bloom filters and index dictionaries) across the distributed system before queries are executed. This preliminary indexing action enables sophisticated searches including JOIN operations and regex patterns to be performed efficiently on petabytes of data, as the index structures are already in place to guide the query execution without scanning entire datasets.
Solution Approach 2:
The patent introduces index structures as intermediary layers between the query interface and the raw data partitions. These index structures (Bloom filters for existence checking and index dictionaries for location mapping) mediate between the query requirements and the distributed data storage, enabling sophisticated search functionality while maintaining query efficiency across petabytes of data.
Data Source
AI summary
Computer-implemented methods and systems are disclosed for receiving and indexing a plurality of files for later querying, for dynamically generating scripts to be executed during a query of a data store, and for horizontally distributing a query and aggregating results of the distributed query.