Real-Time True-False Indexing for In-Memory Structured Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional query processors struggle with processing large volumes of data in memory, requiring significant resources and time to execute queries, and often limit dataset size or complexity to manage memory usage.
Innovation Solution
An in-memory query processor generates and processes structured queries by creating true and false indices within hierarchical datasets, reducing the need for repetitive scanning and memoizing results, thus optimizing memory usage and query execution time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional query processors scan large in-memory datasets, then query results can be obtained, but processing time increases significantly and memory resources are heavily consumed
Solution Approach 1:
The system performs preliminary actions by pre-computing and storing aggregate statistics (count, sum, min, max, distinct count) for each column during data loading. These pre-computed statistics are stored in memory and used to quickly evaluate query conditions without scanning entire tables, thereby reducing query processing time while maintaining result accuracy.
Solution Approach 2:
The patent introduces statistical aggregates as intermediary structures between the raw data and query processing. These aggregates serve as mediators that enable quick evaluation of query conditions (e.g., filtering, grouping, aggregation) without directly accessing or scanning the underlying large datasets, thus reducing processing time while preserving query result correctness.
2Quantity of substance
If conventional query processors handle large datasets, then comprehensive data analysis is possible, but memory resources are excessively consumed
Solution Approach 1:
The system extracts only the essential statistical information (count, sum, min, max, distinct count) from the large datasets and stores these compact representations in memory. This extraction allows the system to handle large datasets by retaining only the minimal necessary data for query processing, dramatically reducing memory consumption while preserving analytical capabilities.
Solution Approach 2:
The patent transforms the data representation from storing complete raw records to storing aggregated statistical parameters (count, sum, min, max, distinct count). This parameter transformation changes the memory footprint from proportional to dataset size to proportional to the number of columns and tables, enabling efficient handling of large datasets with limited memory resources.
3Ease of operation
If conventional tools access individual records, then specific field values can be retrieved, but filtering and summarization operations are difficult
Solution Approach 1:
The patent creates a universal statistical aggregate structure that serves multiple functions: it enables individual record access through the underlying data, supports filtering operations using pre-computed statistics, enables summarization through aggregate functions, and facilitates grouping operations. This multi-functional design allows the same data structure to handle diverse query types without requiring separate processing mechanisms.
Data Source
AI summary
In various embodiments, a computer-implemented method comprises receiving a structured query for an in-memory dataset, identifying, based on the structured query, a plurality of tables included in the in-memory dataset, generating, for each table in the plurality of tables, a first index of records in the table that is associated with at least one field value responsive to the structured query, and a second index of records in the table that is not associated with at least one field value responsive to the structured query, and executing the structured query by processing indices in one or more of the tables to identify a list of records containing field values that are responsive to the structured query.