Zone Maps for Early Exit from Table Scans
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large database table searches in distributed data warehouses face high computational workload due to unsorted data and network overhead from indexing, especially when scanning for query results, as existing methods do not effectively reduce I/O operations and latency.
Innovation Solution
The use of zone maps to sort interval values for determining the scanning order of storage regions, allowing for early exit from table scans when the specified quantity of results is met, thereby reducing computational demands and disk scan time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If indexing is used to provide sorted order effect, then data retrieval efficiency is improved, but network overhead increases and latency worsens
Solution Approach 1:
The patent divides the data warehouse into multiple storage regions (segments) and creates zone maps for each region. These zone maps contain interval values that represent the range of data values in each region. By segmenting the data and maintaining metadata about each segment, the system can avoid centralized indexing while still enabling efficient query processing through regional metadata comparison.
Solution Approach 2:
The patent pre-computes and stores interval values (minimum and maximum data values) for each storage region in zone maps before query execution. This preliminary action allows the system to quickly determine which regions need to be scanned by comparing query conditions against pre-stored interval values, eliminating the need for runtime indexing operations and reducing network overhead.
2Ease of manufacture
If data is stored in unsorted form to avoid sorting cost, then storage flexibility is improved, but query scanning workload increases
Solution Approach 1:
The system performs preliminary computation of interval values (minimum and maximum data values) for each storage region and stores them in zone maps. This allows unsorted data to be stored flexibly while still enabling efficient query evaluation by comparing query conditions against pre-computed interval values to determine which regions need scanning.
Solution Approach 2:
The patent introduces zone maps as an intermediary structure between unsorted data storage and query processing. The zone maps contain interval values that mediate between the unsorted data and the query conditions, allowing the system to maintain storage flexibility while improving query evaluation efficiency by filtering out regions that cannot contain matching results.
3Loss of time
If zone maps are used to determine scanning order, then disk scan time is reduced, but metadata processing complexity increases
Solution Approach 1:
The patent changes the parameters stored in zone maps from detailed data distributions to simplified interval values (minimum and maximum data values) for each storage region. This parameter simplification reduces metadata processing complexity while still enabling effective determination of scanning order by comparing query conditions against these simplified interval parameters.
Data Source
AI summary
According to embodiments of the present invention, methods, systems and computer-readable media are presented for scanning a plurality of storage regions within memory for a specified quantity of results, wherein each storage region is associated with an interval including first and second interval values indicating a value range for values within that storage region. The techniques comprise sorting the first interval values into an order, wherein the order of the first interval values determines a scanning order for the plurality of storage regions, determining a result value, wherein the result value is an upper bound, a lower bound, or is outside of the specified quantity of results, and examining the sorted first interval values and scanning corresponding individual storage regions in response to a comparison of the determined result value with the first interval value of that storage region. In some embodiments, scanning exits upon establishing that the specified quantity of results has been achieved and that a first interval value in a list of sorted first interval values is greater than the determined result value.