Columnar Data Compression with Tuple Map Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression methods for column data in page formats face challenges in efficient query processing due to high overhead in decompression and access times, especially when dealing with variable-length data and sparse data distributions.
Innovation Solution
The method employs order-preserving frequency-based dictionary compression, packing data in a dense bit-aligned manner to enable SIMD processing, along with a tuple map and cumulative histogram for efficient indexing and access, allowing for simultaneous processing of multiple values and reducing bit shifting during access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is compressed using traditional methods, then storage space is reduced, but query processing speed deteriorates due to high decompression overhead
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing cumulative histograms and tuple maps during data compression. These pre-computed structures enable fast query processing without requiring full decompression, as the indexing information is already prepared in advance to guide selective data access and processing.
Solution Approach 2:
The patent introduces cumulative histograms and tuple maps as intermediary structures between the compressed data and query processing operations. These intermediaries provide indexing and statistical information that enable efficient query execution on compressed data without complete decompression, acting as a bridge between space-efficient storage and speed-efficient access.
2Adaptability or versatility
If data is packed in sparse format to handle variable-length data, then data flexibility is improved, but access efficiency deteriorates due to increased bit shifting operations
Solution Approach 1:
The patent applies local quality by using dense bit-aligned packing for regions of data with similar characteristics (e.g., same length prefix) while maintaining overall flexibility through the tuple map structure. Each data region can be optimized independently with appropriate packing density, reducing unnecessary bit shifting operations while still handling variable-length data effectively.
Solution Approach 2:
The patent changes the parameter of data packing from sparse to dense bit-aligned format, fundamentally altering how variable-length data is stored. By using length-prefix encoding with dense packing and complementing it with cumulative histograms that track data distribution, the system achieves both flexibility for variable-length data and efficiency by minimizing bit shifting during access operations.
3Productivity
If tuple map is used for indexing, then data access efficiency is improved, but memory overhead increases
Solution Approach 1:
The patent applies dimensionality change by organizing the tuple map and cumulative histogram as multi-dimensional indexing structures. Instead of a simple linear index, the system uses hierarchical organization where cumulative histograms provide statistical summaries at different levels, enabling the tuple map to be more compact while maintaining efficient access paths through multiple dimensions of organization.
Solution Approach 2:
The patent creates a composite indexing structure combining tuple maps with cumulative histograms. This composite approach integrates two different indexing strategies into a unified structure that achieves better space efficiency than either component alone, while maintaining the access efficiency benefits of tuple maps. The cumulative histogram component provides statistical guidance that reduces the size requirements of the tuple map.
Data Source
AI summary
The embodiments described herein relate to accessing a plurality of data elements. A page of column data is compressed and stored in a format that includes a collection of data elements. A tuple map is stored, and the collection of data elements is indexed via the tuple map. A query is processed based on the compressed page by identifying a set of tuple identifiers mapping to stored data in support of the query. Each tuple identifier corresponds to a location of a respective tuple of the compressed page.