Metadata Table Compression Using Rounded Min-Max and Bloom Filters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data compression techniques, particularly lossy compression methods, face challenges in efficiently reducing data size while maintaining usability, especially in applications requiring real-time communication and storage optimization.
Innovation Solution
The method involves adjusting metadata in tables by determining similarity of topics and using lossy compression on minimum, maximum, and Bloom filter values in metadata tables to create an in-memory structure that monitors predicate usage, thereby optimizing storage and read costs by compressing data in index tables.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If lossy compression is applied to metadata tables, then data storage capacity is reduced, but data accuracy deteriorates
Solution Approach 1:
The patent changes the precision parameter of metadata values dynamically. Minimum and maximum values are stored with reduced precision (fewer decimal places), while Bloom filter values use rounded integers. This parameter adjustment enables significant space reduction in metadata tables while maintaining sufficient accuracy for query operations, resolving the contradiction between storage efficiency and data accuracy.
2Quantity of substance
If metadata table size is reduced through compression, then storage costs decrease, but query performance may deteriorate
Solution Approach 1:
The patent applies different compression strategies to different metadata values based on their local characteristics. Minimum and maximum values use rounded decimal representation, while Bloom filter values use integer rounding. This localized quality adjustment ensures that each metadata type is compressed appropriately for its specific use case, maintaining query performance while reducing overall metadata size.
3Quantity of substance
If Bloom filter values are rounded to integers, then space efficiency is improved, but false positive rate increases
Solution Approach 1:
The patent applies partial rounding to Bloom filter values by rounding to the nearest integer rather than truncating or using full precision. This partial action approach achieves significant space reduction while maintaining acceptable reliability for most query workloads, balancing the trade-off between space efficiency and false positive rate.
Data Source
AI summary
The method includes identifying at least one of a minimum value, a maximum value, and a Bloom filter value for a row of data in a metadata table, wherein the metadata table contains metadata corresponding to a row of data in a main table. The method includes adjusting at least one of an identified first minimum value to a second minimum value, an identified first maximum value to a second maximum value, and an identified first Bloom filter value to a second Bloom filter value.