Flash Cache Dynamic Data Format Rewriting for Query Acceleration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face challenges in balancing CPU and I/O loads due to different access requirements for row-major and column-major data formats, leading to inefficiencies and high costs in maintaining optimal data access, such as in the fractured-mirror and Hybrid Columnar Compression approaches.
Innovation Solution
A method is implemented where data blocks in a storage device are rewritten from a base format to a dynamically generated alternate format, such as column-major format, and stored in a flash cache to optimize access based on query patterns, using metadata and user-rewrite criteria to determine the most efficient format for storage and retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is stored in both row-major and column-major formats (fractured-mirror approach), then query processing performance is improved, but storage cost and implementation complexity increase
Solution Approach 1:
The system performs preliminary actions by proactively rewriting data blocks from row-major to column-major format and caching them in flash storage before they are actually needed for queries. This anticipatory approach ensures that both formats are available when queries arrive, improving query performance without requiring the complex fractured-mirror architecture of maintaining both formats simultaneously across all data.
Solution Approach 2:
The invention introduces flash storage as an intermediary layer between the primary storage and query processing system. This intermediary cache stores rewritten column-major format data blocks, mediating between the row-major data on primary storage and the query requirements, thereby avoiding the need for complex dual-format storage management while still enabling fast query processing.
2Speed
If data is rewritten to alternate format and stored in flash cache, then query access speed is improved, but storage device resources are consumed
Solution Approach 1:
The system dynamically changes parameters by monitoring query patterns and adjusting the rewriting decisions based on observed access patterns. Data blocks are rewritten to column-major format and cached in flash storage when query patterns indicate frequent access to specific columns, and the flash cache is evicted when patterns change or capacity is needed, thereby optimizing the balance between query speed and flash cache utilization.
3Productivity
If data blocks are rewritten dynamically based on query patterns, then data access efficiency is improved, but processing overhead increases
Solution Approach 1:
The system implements self-service by using the query processing system itself to generate the rewriting signals. When the query optimizer determines that a query would benefit from columnar data, it signals the storage system to rewrite the relevant data blocks, eliminating the need for external monitoring systems or complex rewrite decision logic.
Data Source
Figure 1
Figure 2
Figure 3A
AI summary
A method for accelerating queries using dynamically generated columnar data in a flash cache is provided. In an embodiment, a method comprises a storage device receiving a first request for data that is stored in the storage device in a base major format in one or more primary storage devices. The storage device comprises a cache. The base major format is any one of: a row-major format, a column-major format and a hybrid-columnar format. Based on first one or more criteria, it is determined whether to rewrite the data into rewritten data in a rewritten major format. In response to determining to rewrite the data into rewritten data in a rewritten major format, the storage device rewrites at least a portion of the data into particular rewritten data in the rewritten major format. The rewritten data is stored in the cache.