Vector Architecture Query Processing with Bloom Filter Bitmasks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for processing complex data queries in 'Big Data' stored in relational databases are inefficient, as they do not effectively leverage GPU capabilities for parallelized processing, limiting the ability to handle large datasets efficiently.
Innovation Solution
A method and system that utilize enhanced data structure representations, including logical data chunk boundaries and Bloom filter bitmasks, to enable simultaneous search across multiple data chunks using HWA processors, allowing for efficient query processing by identifying data item addresses and executing queries on respective data chunks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data queries are processed in parallel using known parallel computational processing methods, then processing capability is improved, but the methods only solve specific tailored scenarios and do not effectively leverage GPU capabilities for general Big Data queries
Solution Approach 1:
The patent segments Big Data into multiple data chunks and organizes them in a tree-like hierarchical structure with chunk groups and chunks. This segmentation enables parallel processing of different chunks while maintaining the ability to handle various query types. Each chunk is independently processable by GPU threads, achieving both parallel productivity and versatility across different query scenarios.
2Productivity
If enhanced data structure representation with logical data chunk boundaries and Bloom filter bitmask is used, then search capability and compression ratio are improved, but data structure complexity increases
Solution Approach 1:
The patent pre-computes and stores Bloom filter bitmasks and logical data chunk boundaries during data loading and preprocessing stages. This preliminary action enables fast search operations during query execution without computing these structures in real-time. The enhanced data structure is built once and reused, improving search efficiency while the complexity is paid for only during the initial setup phase.
3Speed
If simultaneous search across multiple data chunks is performed using HWA processors, then processing speed is improved, but memory requirements and computational overhead increase
Solution Approach 1:
The patent uses Bloom filter bitmasks to perform partial filtering before full data chunk processing. The Bloom filter provides a probabilistic first layer of filtering that quickly eliminates chunks that definitely don't match the query, allowing HWA processors to focus only on relevant chunks. This partial action approach speeds up search while reducing the effective memory and computational load on the expensive HWA processing stage.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The present invention provides a method for pre-processing and processing query operation on multiple data chunk on vector enabled architecture. The method comprising the step of : receiving user query having at least one a data item, accessing data chunk blocks having enhanced data structure representation, wherein the enhanced data structure representation includes data recursive presentation of data chunk boundaries and bloom filter bitmask of data chunks, search simultaneously at multiple data chunk blocks utilizing the recursive presentation of data chunk boundaries using HWA, identifying data item address by comparing calculated Bloom filter bitmask of the requested data item to calculated bitmask of the respective data chunks simultaneously by using multiple HWA, executing query on respective data chunk.