Selective Data Decompression for Database Query Optimization

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

As databases grow, managing compressed and uncompressed data efficiently becomes challenging, as constant compression and decompression reduce query throughput and increase resource usage, leading to slower query execution times.

Innovation Solution

Implementing a method to selectively uncompress data based on uncompression criteria, such as system conditions and query types, allowing for partial uncompression or ignoring compressed data to prioritize faster query response times, thereby optimizing memory usage and resource allocation.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Quantity of substance

If data compression is applied to reduce paging and improve query execution time, then memory usage efficiency is improved, but query throughput deteriorates due to additional compression and decompression operations

Engineering Contradiction:

Improvememory usage efficiencyVSAvoidquery throughput

Core Design Contradiction:

Quantity of substanceVSProductivity

Solution Approach 1:

The patent applies local quality by differentiating the treatment of compressed and uncompressed data based on their location and accessibility. Compressed data is stored in a compressed data structure with associated metadata, while uncompressed data is stored separately. The system selectively decompresses only the compressed portions needed for a query rather than decompressing all data, thereby improving query throughput while maintaining memory efficiency for frequently accessed uncompressed data.

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The patent implements preliminary action by pre-storing uncompressed data in memory alongside compressed data structures. This allows the system to immediately provide uncompressed data for frequently accessed records without requiring decomposition operations at query time, thus maintaining high query throughput while still utilizing compression for less frequently accessed data.

Inventive Principle:

Principle #10Preliminary action

2Reliability

If all compressed data is decompressed to ensure complete data availability, then data accessibility is improved, but processing time increases due to unnecessary decompression operations

Engineering Contradiction:

Improvedata accessibilityVSAvoidprocessing time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent applies partial action by decompressing only the necessary portions of compressed data required by the query rather than decompressing all compressed data. The system uses metadata to identify which compressed data segments are needed and selectively decompresses only those, thereby maintaining data accessibility for requested records while minimizing processing time by avoiding unnecessary decompression operations on unused data.

Inventive Principle:

Principle #16Partial or excessive action

Solution Approach 2:

The patent extracts the necessary information from compressed data structures using associated metadata before decomposition. The metadata contains pointers and indices that allow the system to identify and extract only the specific compressed data segments needed for the query, enabling selective decompression and reducing processing time while ensuring complete data availability for the requested records.

Inventive Principle:

Principle #2Taking out (Extraction)

3Quantity of substance

If compressed data is stored to reduce memory footprint, then storage efficiency is improved, but query execution time increases due to decomposition requirements

Engineering Contradiction:

Improvestorage efficiencyVSAvoidquery execution time

Core Design Contradiction:

Quantity of substanceVSLoss of time

Solution Approach 1:

The patent segments data storage into distinct compressed and uncompressed data structures, each with its own metadata. This segmentation allows the system to store data in compressed form for efficiency while maintaining the ability to quickly access and decompress only the necessary segments on demand, thereby reducing query execution time compared to fully compressed systems while maintaining storage efficiency.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces metadata as an intermediary between the compressed data structure and the query processing system. This metadata contains information about the compressed data organization, pointers to relevant segments, and decomposition instructions, enabling the system to efficiently navigate and decompress only the necessary data portions without requiring full decomposition, thus reducing query execution time while maintaining storage efficiency.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS8554747B2Active memory expansion in a database environment to query needed/uneeded results

Publication Date: 2013.10.08 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

Techniques are described for estimating and managing memory compression for request processing. Embodiments of the invention may generally include receiving a request for data, determining if the requested data contains any compressed data, and sending the requesting entity only the uncompressed data. A separate embodiment generally includes receiving a request for data, determining if the requested data contains any compressed data, gathering uncompression criteria about the requested data, and using the uncompression criteria to selectively determine what portion of the compressed data to uncompress.