Metadata-Verified Data Caching for Consistent Reads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data caching systems face challenges in ensuring strong consistency, high performance, and high availability, particularly in big data analyses, due to reliance on outdated update times and node dependencies, leading to inconsistencies and reduced system availability.
Innovation Solution
A data caching method that verifies consistency by using metadata from a data storage system to determine whether cached data matches the stored data, allowing for timely and accurate caching and reading, while maintaining system availability and compatibility with direct access paths.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a distributed caching system is established to improve data reading efficiency, then the efficiency of data reading is improved, but the requirements for strong consistency, high performance, and high availability bring great challenges to the global design
Solution Approach 1:
The system segments the data access architecture into three independent layers: the client layer that initiates requests, the caching system layer that handles consistency verification using metadata, and the storage system layer that provides persistent storage. This segmentation allows each layer to operate independently, reducing global design complexity while maintaining high reading efficiency.
Solution Approach 2:
The caching system acts as an intermediary between clients and the storage system. It verifies data consistency by comparing metadata (such as cyclic redundancy check values and update times) without requiring clients to directly interact with the storage system, thereby simplifying the overall design while ensuring consistency and high performance.
2Productivity
If cached data is used to accelerate big data analyses, then the utilization rate of clusters is improved, but hot topic data occupies most of the bandwidths provided by the object storage services
Solution Approach 1:
The system performs preliminary actions by caching frequently accessed data in the caching system before it needs to be read from the storage system. This allows hot data to be served from the cache, reducing the bandwidth occupation of the storage system while improving cluster utilization for big data analyses.
Solution Approach 2:
The caching system creates copies of hot data from the storage system and stores them in memory. These copies serve subsequent read requests, reducing the bandwidth burden on the storage system while maintaining high cluster utilization for analytical workloads.
3Measurement precision
If metadata verification is performed to ensure data consistency, then the accuracy of cached data is improved, but the time required for consistency determination increases
Solution Approach 1:
The system extracts only the essential metadata elements (such as cyclic redundancy check values and update times) needed for consistency verification, rather than verifying the entire data set. This extraction approach maintains high accuracy in consistency determination while minimizing the time required for verification.
Solution Approach 2:
The system changes the verification parameters from full data comparison to metadata attribute comparison (cyclic redundancy check values, update times). This parameter change maintains measurement precision for consistency while dramatically reducing the time required for verification.
Data Source
AI summary
A data caching and reading method, and a data access system are provided. A client sends a first request to a data storage system so as to obtain metadata of target data. The client sends a second request to a data caching system, the second request comprising the metadata. The data caching system determines, on the basis of the metadata, whether cache target data consistent with the target data stored in the data storage system is cached. When it is determined cache target data is cached, the client obtains the target data from the data caching system. When no consistent cache target data is cached, the client obtains the target data from the data storage system, and sends the obtained target data to the data caching system, so as to cache the target data.