Metadata Table Compression for Lower Read Cost and Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data compression techniques, particularly lossy compression methods, face challenges in efficiently reducing data size while maintaining usability, especially in applications requiring real-time communication and storage optimization.
Innovation Solution
The method involves adjusting metadata in tables by determining similarity of topics and using lossy compression on minimum, maximum, and Bloom filter values in metadata tables to create an in-memory structure that monitors predicate usage, thereby optimizing storage and read costs by eliminating unnecessary data retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If lossy compression is applied to metadata tables, then storage space is reduced and data retrieval efficiency is improved, but data precision and completeness are degraded
Solution Approach 1:
The patent applies different compression strategies to different parts of the metadata table based on their importance and usage patterns. Frequently accessed metadata is compressed with lower aggression, while less frequently accessed metadata undergoes more aggressive compression. This local differentiation allows the system to optimize storage space while preserving precision for the most critical data elements.
Solution Approach 2:
The system dynamically adjusts compression parameters based on metadata characteristics and access patterns. By changing compression ratios, data types, and storage formats according to specific metadata properties, the system achieves optimal balance between storage reduction and data precision maintenance for different metadata segments.
2Volume of stationary object
If aggressive lossy compression is applied to reduce data size, then storage capacity requirements are reduced, but data usability and query accuracy deteriorate
Solution Approach 1:
The patent applies partial compression only to non-critical metadata fields while leaving critical fields uncompressed or lightly compressed. This selective approach ensures that data usability is maintained for essential query operations while achieving storage reduction for less important metadata, avoiding the pitfalls of aggressive full-table compression.
Solution Approach 2:
The system introduces intermediary compression levels and hybrid storage formats that mediate between fully compressed and fully uncompressed states. These intermediary representations allow the system to achieve moderate storage reduction while preserving sufficient data usability for most query operations, serving as a compromise between extreme compression and full precision storage.
3Quantity of substance
If metadata is compressed to improve storage efficiency, then storage costs are reduced, but the complexity of data management and decompression processes increases
Solution Approach 1:
The patent pre-compresses metadata during the data loading and ETL processes rather than compressing on-the-fly during queries. This preliminary compression action eliminates the need for complex real-time decompression operations during data retrieval, reducing query complexity while maintaining storage efficiency benefits. The compression work is performed once during data preparation rather than repeatedly during access.
Data Source
AI summary
The method includes identifying at least one of a minimum value, a maximum value, and a Bloom filter value for a row of data in a metadata table, wherein the metadata table contains metadata corresponding to a row of data in a main table. The method includes adjusting at least one of an identified first minimum value to a second minimum value, an identified first maximum value to a second maximum value, and an identified first Bloom filter value to a second Bloom filter value.