Key-Value Storage System with Hash-Based Dual-Layer Persistence
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current storage systems face challenges in achieving high-speed and reliable data storage while ensuring scalability, as volatile memory requires frequent data transfer to non-volatile storage, leading to increased response times and reduced performance.
Innovation Solution
A key-value storage system is implemented, where data is stored using hash values to reduce the amount of data written to disk, utilizing a two-stage indirect reference configuration to differentiate between volatile and non-volatile storage, allowing for efficient data retrieval and error detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored only in volatile memory to achieve high-speed processing, then access speed is improved, but data reliability deteriorates because data is lost when power is off
Solution Approach 1:
The storage system is segmented into two distinct layers: volatile memory for high-speed data access and non-volatile storage for data persistence. The patent divides data storage functionality between these two media, allowing each to operate in its optimal performance zone while working together as a unified system.
Solution Approach 2:
A storage management unit acts as an intermediary between the volatile memory and non-volatile storage. This mediator selectively transfers data between the two media based on access patterns, using techniques like write-back caching and data pre-loading to maintain high-speed access while ensuring data is eventually persisted to non-volatile storage.
2Reliability
If data is frequently transferred from volatile memory to non-volatile storage to ensure reliability, then data persistence is improved, but access speed deteriorates due to increased disk access time
Solution Approach 1:
The system performs preliminary actions by pre-loading data into volatile memory before it is actually needed for processing. The storage management unit anticipates data access patterns and proactively transfers data from non-volatile storage to volatile memory, so that when applications need the data, it is already available in high-speed memory.
Solution Approach 2:
Data synchronization between volatile memory and non-volatile storage occurs periodically rather than continuously. The storage management unit uses techniques like write-back caching where data is modified in volatile memory and then periodically or asynchronously flushed to non-volatile storage, reducing the frequency of disk access operations.
3Stability of the object's composition
If traditional storage architectures are used with clear separation between volatile and non-volatile storage, then system stability is improved, but scalability deteriorates in distributed environments
Solution Approach 1:
The storage management unit implements a universal interface that works across both volatile and non-volatile media, abstracting the underlying differences. This allows the same data access patterns and storage operations to function uniformly whether data resides in high-speed memory or persistent storage, enabling seamless integration in distributed systems.
Solution Approach 2:
The patent transitions from a single-dimension storage model (either volatile or non-volatile) to a two-dimensional model that simultaneously utilizes both media types. By adding the dimension of persistence alongside the dimension of speed, the system achieves both high-performance access and data durability, while the abstracted interface enables scalable deployment across distributed nodes.
Data Source
AI summary
A system has a data structure in which a value can be obtained from a key. In a write access, a first pair <Key,Hash(Value)> and a second pair <Hash(Value),Value> are stored respectively in a volatile storage device. The first pair <Key,Hash(Value)> is saved in a nonvolatile storage device before returning a response, and the second pair <Hash(Value),Value> is saved in the first storage device at any time with the second pair saved in the volatile storage device. In a read access in which a value is obtained from a key, it is determined that data is not stored normally if the second pair is not found in processing in which after obtaining the hash value of the value from the first pair, the second pair is read.