Persistent Data Structure Storage Without HDD Write-Back Delays
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage systems using hard disk drives (HDDs) suffer from slow access times, and caching solutions require synchronization with permanent storage, while conventional data management methods like deduplication are inconvenient for operations on data blocks.
Innovation Solution
Implementing a data structure store management system that utilizes non-volatile random access memory devices, such as ReRAM and NAND Flash, to store data in a format that allows for faster and more reliable access, using erasure coding and metadata management to maintain data integrity and facilitate operations like push, pop, and search on data structures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If hard disk drives (HDDs) are used for storage, then cost efficiency and reliability are improved, but access speed deteriorates
Solution Approach 1:
The storage system is segmented into two distinct layers: a fast but expensive volatile memory layer (DRAM) for frequently accessed data, and a slow but cheap non-volatile storage layer (HDD) for permanent storage. This segmentation allows the system to achieve fast access speeds for active data while maintaining the reliability and cost-efficiency of HDD-based permanent storage.
Solution Approach 2:
A volatile memory buffer (DRAM) is introduced as an intermediary between the client and the HDD storage. This buffer temporarily holds data during transfer operations, enabling fast write-back to HDDs and quick read delivery to clients, thereby resolving the speed limitation of HDDs while maintaining their reliability.
2Speed
If cache memory devices are used to accelerate data access, then access speed is improved, but system complexity increases due to synchronization requirements
Solution Approach 1:
The system employs write-back caching where the cache automatically manages itself by tracking which data blocks have been modified. When data is written to the cache, it is marked as dirty and will be automatically written back to the HDD when the cache buffer is full or when data is evicted, eliminating the need for complex application-level synchronization logic.
Solution Approach 2:
The patent replaces the traditional mechanical block-based cache management with a software-managed page-based caching system that uses in-memory data structures to track cache state. This substitution of mechanical synchronization with software-based management simplifies the overall system architecture while maintaining fast access performance.
3Quantity of substance
If data is stored in conventional block format on HDDs, then storage capacity is maximized, but data management operations become inconvenient
Solution Approach 1:
The system introduces a logical dimension above the physical block storage by implementing a virtual file system that maps logical data structures (files, directories, metadata) onto the physical block devices. This allows data management operations to be performed on logical entities rather than raw blocks, greatly improving ease of operation while maintaining efficient use of storage capacity through features like sparse files and dynamic allocation.
Data Source
AI summary
The instant disclosure provides a data structure store system and a method of managing data in the store. The method includes receiving, by a data structure store management system, a request for storing data from a client. In the created data structure, each data element includes a portion of the data. On receiving a read request for at least part of the data, the data structure store management system provides at least part of the data to a recipient device. The data elements are stored in persistent memory in the form of one or more non-volatile random access devices, wherein during a time interval between receiving the storing request and providing the at least part of the data by the data structure store management system to the recipient device, the data structure store management system provides no portion of the data for writing to a hard disk drive.