In-Memory Dataset Indexing for Faster Structured Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional systems face challenges in efficiently processing and aggregating large volumes of data from in-memory datasets, requiring significant processing resources and time due to the need for intense memory usage and slow query processing.
Innovation Solution
A computer-implemented method that generates first and second indices for records in a structured query, allowing efficient processing of in-memory datasets by identifying responsive and non-responsive records, thereby reducing memory resources and query time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional query processors are used to search in-memory datasets, then query results can be obtained, but significant processing resources and time are required
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing indices for records in the in-memory dataset before queries are executed. These indices organize data by field values and relationships, allowing the query processor to quickly locate relevant records without scanning entire tables. This pre-prepared structure enables fast query execution while maintaining accurate results.
2Quantity of substance
If the size of in-memory datasets is increased to store more records, then more data can be accessed, but memory resources are exhausted
Solution Approach 1:
The patent applies segmentation by dividing the in-memory dataset into multiple tables organized by specific field values and relationships. Instead of storing all records in a single large structure, the data is segmented into manageable tables that can be independently indexed and queried. This segmentation reduces the memory footprint required for any single query operation while enabling access to large numbers of records across multiple tables.
3Ease of operation
If conventional tools are used to access individual records, then simple retrieval is possible, but filtering and summarizing operations are difficult
Solution Approach 1:
The patent applies universality by designing a table structure and indexing system that serves multiple functions simultaneously. The same indexed table structure that enables simple record retrieval also supports filtering operations by field value, summarizing operations by grouping, and complex query operations. This multi-functional design eliminates the need for different tools for different operations while maintaining ease of use.
Data Source
AI summary
In various embodiments, a computer-implemented method comprises receiving a structured query for an in-memory dataset, identifying, based on the structured query, a plurality of tables included in the in-memory dataset, generating, for each table in the plurality of tables, a first index of records in the table that is associated with at least one field value responsive to the structured query, and a second index of records in the table that is not associated with at least one field value responsive to the structured query, and executing the structured query by processing indices in one or more of the tables to identify a list of records containing field values that are responsive to the structured query.