Columnar Storage Warehouses with Key-Value Buffering for Real-Time Reads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing columnar storage data warehouses experience a significant time delay between data generation and reading, which hinders real-time data analysis capabilities.
Innovation Solution
A data processing method that involves storing data in a key-value storage module for small data writes and a columnar storage module for larger writes, utilizing key-value pairs to manage data locks and batch transfer to columnar storage for efficient real-time data access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored in a columnar storage module using columnar storage technology, then storage efficiency for large datasets is improved, but time delay increases and real-time access capability deteriorates
Solution Approach 1:
The patent divides the storage system into two separate modules: a key-value storage module for real-time small data writes and a columnar storage module for efficient large data storage. This segmentation allows each module to be optimized for its specific function, resolving the contradiction between real-time access and bulk storage efficiency
Solution Approach 2:
The key-value storage module acts as an intermediary between data generation sources and the columnar storage module. It temporarily holds small data writes in real-time and batches them for transfer to columnar storage, mediating between the conflicting requirements of immediate access and efficient bulk storage
2Speed
If data is stored in a key-value storage module for real-time access, then real-time analysis capability is improved, but storage efficiency for large datasets deteriorates
Solution Approach 1:
The storage system is segmented into two specialized modules: key-value storage optimized for fast real-time access to small datasets, and columnar storage optimized for efficient handling of large datasets. Each segment excels at its specific function without compromise
Solution Approach 2:
The system uses partial action by storing only the necessary portion of data in the key-value module for real-time access, while batching the remainder for columnar storage. This avoids the excessive overhead of using key-value storage for all data sizes
3Productivity
If small data writes are handled separately from large data writes, then real-time processing of small writes is improved, but system complexity increases
Solution Approach 1:
The key-value storage module serves multiple functions: it handles real-time small data writes, maintains data locks for concurrency control, and acts as a batching buffer for columnar storage transfers. This multi-functionality reduces the need for separate specialized components
Solution Approach 2:
The system uses the key-value storage module's own structure to manage data locks and track write operations, rather than requiring separate lock management infrastructure. The key-value pairs themselves encode lock state information, enabling self-service concurrency control
Data Source
AI summary
A data processing method, an electronic device, and a storage medium. The data processing method includes: receiving a data write request; and in response to the target write manner being a first write manner, storing the target data in a key-value storage module by using a key-value storage manner, so that the target data is subsequently read from the key-value storage module, each row of data in the at least one row of data respectively corresponds to one key-value pair, and a data write amount corresponding to the first write manner is less than a first threshold; or, in response to the target write manner being a second write manner, storing the target data in a columnar storage module by using a columnar storage technology, a data write amount corresponding to the second write manner is greater than a data write amount corresponding to the first write manner.