Spillover Slot Cache Memory Data Replication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data replication systems, such as synchronous RDF and asynchronous SAR, face challenges in balancing latency and data consistency across distant storage devices, with synchronous modes experiencing high latency and asynchronous modes risking data inconsistency and significant overhead.
Innovation Solution
A system that manages data by segregating writes into chunks, transferring them after completion, and using cache memory efficiently, with data stored on local disks when memory thresholds are exceeded, ensuring low latency and data consistency through chunk-based data management and efficient cache utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If synchronous RDF data transfer is used, then data consistency between primary and secondary storage devices is ensured, but latency increases significantly especially over long distances
Solution Approach 1:
The patent segments data into fixed-size chunks (e.g., 64KB) and processes them independently. Each chunk is assigned a sequence number and can be transferred separately, allowing parallel processing and reducing the impact of long-distance transmission delays on overall latency.
Solution Approach 2:
The system performs preliminary actions by pre-allocating chunk buffers and preparing data for transfer before actual transmission begins. The host adapter pre-processes data into chunks and queues them for transfer, reducing the critical path latency during actual data operations.
2Loss of time
If asynchronous background copy is used, then latency is reduced, but data consistency and reliability are compromised
Solution Approach 1:
The patent implements a feedback mechanism where the host adapter tracks the transfer status of each chunk and receives acknowledgments from the remote storage device. This feedback ensures that data consistency is maintained while allowing asynchronous operation, as the system can detect and recover from transmission failures.
Solution Approach 2:
The system performs preliminary actions by pre-processing data into chunks with sequence numbers and metadata before transfer. This preparation enables reliable asynchronous operation because the receiving端 can reconstruct data in the correct order and detect missing or corrupted chunks.
3Productivity
If cache memory is used to buffer data chunks, then transfer efficiency is improved, but cache memory usage increases
Solution Approach 1:
The patent segments cache memory into fixed-size chunk buffers (e.g., 64KB each) that can be independently managed. This segmentation allows efficient utilization of cache memory by allocating only the number of buffers needed for current transfer operations, rather than requiring large contiguous memory regions.
Solution Approach 2:
The system implements a buffer recycling mechanism where used chunk buffers are returned to a free buffer pool after successful transfer. This allows the same physical memory to be reused for multiple transfer operations, reducing overall cache memory requirements while maintaining high transfer efficiency.
Data Source
AI summary
Managing a cache memory includes providing at least one spillover slot, stored in the cache memory, that contains information about a plurality of cache data slots transferred from the cache memory to a local disk. The spillover slot has a metadata portion with information about cache data slots transferred from the cache memory to the local disk and has a catalog portion with entries that point to different sections of the metadata portion. Each section of the metadata portion corresponding to information about one of the cache data slots transferred from the cache memory to the local disk.