Parallel Block Scan for Database File Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data platforms underutilize processing capability when a single thread is responsible for scanning a file, especially when there are fewer files compared to the number of available execution threads, leading to inefficient processor utilization and increased IO penalties.
Innovation Solution
Implementing a parallel scan technique where multiple threads access and process different subsets of blocks within a single file, allowing each thread to scan its designated block in parallel, thereby increasing processor utilization and reducing IO penalties by downloading data only once.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a single thread scans a file, then the implementation is simple, but processor utilization is underutilized and IO penalties increase
Solution Approach 1:
The patent segments a single file into multiple blocks that can be scanned in parallel by different threads. Each thread is assigned specific blocks to scan, transforming a single-thread sequential scan into a multi-thread parallel scan operation, thereby improving processor utilization without significantly increasing implementation complexity
Solution Approach 2:
The patent introduces parallelism as a new dimension to the file scanning operation. Instead of scanning blocks sequentially in one dimension (time), multiple threads scan different blocks simultaneously in parallel, effectively adding a spatial dimension (multiple threads working concurrently) to the scanning process
2Device complexity
If a single thread scans a file, then the implementation is straightforward, but IO penalties increase
Solution Approach 1:
By segmenting the file into multiple blocks and assigning them to different threads, the system can overlap IO operations with processing operations. While one thread is processing data from its assigned blocks, other threads can simultaneously perform IO operations on their blocks, thereby reducing total IO time and penalties
3Productivity
If multiple threads perform parallel scan of blocks, then processor utilization increases, but the system complexity increases
Solution Approach 1:
The patent divides the file into blocks and assigns specific blocks to specific threads, creating a clear segmentation that simplifies the parallel scanning logic. Each thread independently scans its assigned blocks without complex coordination, reducing the overall system complexity while maintaining high processor utilization
Solution Approach 2:
Each thread independently manages its own block scanning operations without requiring complex inter-thread coordination or synchronization. The threads self-serve by autonomously processing their assigned blocks, which simplifies the overall system architecture and reduces complexity
Data Source
AI summary
Multiple execution threads process a query directed to a database organized into a plurality of files. In processing the query, a first thread downloads a file from the plurality of files. The file comprises a set of blocks. A parallel scan of the set of blocks is performed by at least the first thread and a second thread to identify data that matches the query. A response to the query is provided based in part on the parallel scan of the set of blocks.