Structured Queries on In-Memory Datasets Using Inverted Indices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional systems struggle with efficiently processing large volumes of data from source datasets, requiring intense processing resources and lengthy search times due to the need for scanning large datasets, often limiting the size or complexity of in-memory datasets.
Innovation Solution
A computer-implemented method involving an in-memory query processor that loads multiple in-memory datasets into local memory, executes structured queries on these datasets, and generates query results by building indices to reduce repetitive processing, thereby reducing memory resources and search times.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional query processors scan large source datasets to retrieve information, then search completeness is improved, but processing time and resource consumption increase significantly
Solution Approach 1:
The system performs preliminary actions by scanning source datasets and building inverted indices in advance, storing frequency information and document references. This preliminary index construction enables rapid query responses without rescaning the entire dataset, thus resolving the contradiction between search completeness and processing time.
Solution Approach 2:
The patent introduces an intermediary inverted index structure that mediates between the source dataset and query processor. This index acts as a bridge, allowing the query processor to access pre-processed frequency and reference information without directly scanning the full source dataset, thereby reducing processing time while maintaining search accuracy.
2Speed
If in-memory datasets are used to reduce query response time, then search speed is improved, but memory resource requirements increase
Solution Approach 1:
The system extracts only the necessary information from source datasets to build inverted indices, storing frequency counts and document references rather than loading entire datasets into memory. This extraction approach enables fast query processing while minimizing memory consumption, resolving the contradiction between search speed and memory resources.
Solution Approach 2:
The patent changes the data representation parameters by transforming source dataset records into inverted index structures with frequency and reference fields. This parameter transformation allows the system to maintain fast access speeds through in-memory indexing while reducing the quantity of data stored in memory compared to traditional full-dataset approaches.
3Adaptability or versatility
If source datasets are scanned repeatedly for different queries, then query flexibility is improved, but processing efficiency deteriorates
Solution Approach 1:
The system performs preliminary scanning of source datasets to build inverted indices that capture frequency information and document references. This preliminary action enables the system to handle diverse queries efficiently without rescaning the source data, thus maintaining query flexibility while improving processing efficiency for multiple queries.
Solution Approach 2:
The inverted index structure enables continuous and efficient query processing by maintaining pre-computed frequency and reference information. The system can continuously answer different queries by accessing the index rather than repeatedly scanning source datasets, thus sustaining both query flexibility and processing efficiency over time.
Data Source
AI summary
In various embodiments, a computer-implemented method comprises receiving a structured query identifying a plurality of source datasets, determining, based on a plurality of schemas associated with the plurality of source datasets, the plurality of source datasets associated with the structured query, loading a plurality of in-memory datasets, each in-memory dataset representing a source dataset in the plurality of source datasets, and executing the structured query on each of the plurality of in-memory datasets by causing portions of the structured query to be applied to each of the plurality of in-memory datasets to generate a query result.