Metadata Compression Using Lossy Min-Max and Bloom Filters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data compression techniques, particularly lossy compression, face challenges in efficiently reducing data size while maintaining usability, especially in applications requiring real-time communication and storage optimization.
Innovation Solution
The method involves adjusting metadata in tables by determining similarity of topics and using lossy compression on minimum, maximum, and Bloom filter values in metadata tables to create an in-memory structure that monitors predicate usage, thereby optimizing storage and read costs by eliminating unnecessary data retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If lossy compression is applied to metadata tables, then storage space is reduced and data transmission efficiency is improved, but data precision and query accuracy may be degraded
Solution Approach 1:
The patent changes the precision parameter of metadata values dynamically. For minimum and maximum values, it stores them with reduced precision (fewer significant digits). For Bloom filter values, it adjusts the hash function output precision. This allows the system to achieve compression while maintaining sufficient precision for query operations, resolving the contradiction between storage reduction and data precision.
Solution Approach 2:
The patent applies partial compression by selectively reducing precision only for metadata values that tolerate approximation (minimum, maximum, and Bloom filter values), while maintaining full precision for other critical data. This partial application of lossy compression achieves storage reduction without significantly impacting query accuracy, balancing the contradiction between compression ratio and data precision.
2Speed
If lossy compression is applied to metadata, then data transmission speed is improved, but data quality degradation becomes noticeable
Solution Approach 1:
The patent changes the representation parameters of metadata to enable faster transmission. By storing minimum and maximum values with reduced precision and using compact Bloom filter representations, the data size is significantly reduced, enabling faster transmission. The parameter changes are designed to maintain data quality within acceptable thresholds, resolving the contradiction between transmission speed and data quality.
3Quantity of substance
If metadata is compressed, then storage requirements are reduced, but query processing accuracy may be affected
Solution Approach 1:
The patent applies parameter changes to metadata storage format, using lossy compression for minimum, maximum, and Bloom filter values. This reduces storage requirements while the chosen compression parameters are optimized to minimize impact on query processing accuracy. The system achieves compression ratios of 4:1 to 10:1 while maintaining query accuracy within acceptable bounds.
Solution Approach 2:
The patent applies partial compression only to specific metadata fields (minimum, maximum, Bloom filter) that can tolerate some precision loss, while maintaining full accuracy for other fields. This selective approach reduces overall storage requirements while preserving query processing accuracy for critical operations, resolving the contradiction between compression and query accuracy.
Data Source
AI summary
The method includes identifying at least one of a minimum value, a maximum value, and a Bloom filter value for a row of data in a metadata table, wherein the metadata table contains metadata corresponding to a row of data in a main table. The method includes adjusting at least one of an identified first minimum value to a second minimum value, an identified first maximum value to a second maximum value, and an identified first Bloom filter value to a second Bloom filter value.