In-Memory Database Compression Levels for Faster Data Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Database systems face performance issues due to the need for frequent data compression and decompression in volatile memory, leading to increased compute resource usage and inefficient data management, especially when dealing with large datasets, as not all data can be cached in volatile memory, resulting in significant performance penalties and wasteful resource expenditure from frequent data shuffling.
Innovation Solution
The implementation of a mirror format data system that maintains data in both persistent and in-memory formats, allowing selective data mirroring and compression based on access statistics, data types, and operational needs, enabling faster access to frequently used data while minimizing resource consumption by optimizing data storage and retrieval in volatile memory.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is compressed before storing in volatile memory, then more data can be cached in volatile memory, but frequent decompression consumes compute resources and slows down data operations
Solution Approach 1:
The patent applies different compression levels to different data portions based on their access characteristics. Frequently accessed data is stored at lower compression levels or uncompressed to minimize decompression overhead, while less frequently accessed data uses higher compression levels to maximize cache capacity. This selective approach resolves the contradiction by optimizing the trade-off between cache quantity and operation speed on a per-data-portion basis.
Solution Approach 2:
The patent dynamically adjusts compression levels based on observed access patterns. Data that becomes frequently accessed can have its compression level reduced over time, while data that becomes less accessed can be compressed more aggressively. This dynamic adaptation allows the system to maintain optimal performance characteristics as data usage patterns evolve, balancing cache capacity and operation speed.
2Speed
If volatile memory is filled with cached data, then data access speed improves, but frequent data replacement increases compute resource usage and reduces efficiency
Solution Approach 1:
The patent changes the compression parameter based on data characteristics and access patterns. By adjusting compression levels, the system can control the effective size of cached data, thereby managing cache replacement frequency. Less frequently accessed data is compressed more to fit more diverse data in cache, reducing replacement events, while frequently accessed data maintains lower compression for faster access, resolving the contradiction between access speed and energy loss.
3Quantity of substance
If all data is compressed to fit in volatile memory, then cache capacity increases, but not all data can be significantly compressed leading to limited space savings
Solution Approach 1:
The patent recognizes that different data portions have different compressibility characteristics. Instead of applying a uniform compression approach, it evaluates each data portion's compression potential and applies appropriate compression levels. This localized approach maximizes overall space savings by aggressively compressing compressible data while using alternative strategies for data that resists compression, thereby improving cache capacity without relying solely on compression.
4Quantity of substance
If data is frequently accessed from disk storage, then volatile memory can be kept smaller, but reading from disk storage incurs significant performance penalties
Solution Approach 1:
The patent proactively identifies data that is likely to be accessed based on access patterns and pre-loads it into volatile memory at optimized compression levels before actual access occurs. This preliminary action ensures that frequently accessed data is already in volatile memory when needed, avoiding disk read performance penalties while maintaining efficient use of volatile memory space through appropriate compression selection.
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
Techniques are provided for maintaining data persistently in one format, but making that data available to a database server in more than one format. Data that is in the format that is independent of the disk format may be maintained exclusively in volatile memory to reduce the overhead associated with keeping the data in sync with the on-disk format copies of the data. Selection of data to be maintained in the volatile memory may be based on various factors. Once selected the data may also be compressed to save space in the volatile memory. The compression level may depend on one or more factors that are evaluated for the selected data. The factors for the selection and compression level of data may be periodically evaluated, and based on the evaluation, the selected data may be removed from the volatile memory or its compression level changed accordingly.