NVRAM Buffer for Big Data Analytics Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional storage systems face inefficiencies in data management and storage operations, particularly in handling large data sets and ensuring data integrity and availability, due to redundant write operations and lack of centralized control over storage processes.
Innovation Solution
The implementation of a storage system that utilizes non-volatile random access memory (NVRAM) as a buffer for write operations, offloading device management from storage drives, and employing a direct-mapped flash storage system where the operating system initiates and controls processes, such as data allocation and garbage collection, to optimize storage operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional storage systems perform write operations directly to storage drives, then data is stored, but redundant write operations increase device complexity and reduce productivity
Solution Approach 1:
The patent introduces NVRAM as an intermediary buffer between the host system and storage drives. Write operations are first directed to the NVRAM buffer, which then manages the actual writing to storage drives. This intermediary layer reduces redundant write operations to the drives and simplifies storage management by centralizing control in the buffer layer.
2Quantity of substance
If storage drives perform all write operations, then data is stored, but unnecessary write operations increase loss of time and reduce productivity
Solution Approach 1:
The system performs preliminary write operations to the NVRAM buffer before finalizing storage to the drives. The NVRAM buffer temporarily holds data and manages write operations, allowing the system to batch and optimize writes to storage drives, thereby reducing the total number of write operations and associated latency.
3Ease of operation
If device management is distributed across storage drives, then storage operations are decentralized, but centralized control is needed to optimize storage operations
Solution Approach 1:
The patent merges storage management functions into a centralized NVRAM buffer layer that sits between the host and storage drives. This single point of control manages all write operations, coordinates data flow, and optimizes storage operations, thereby improving productivity while maintaining ease of operation through centralized management.
Data Source
AI summary
Executing a big data analytics pipeline in a storage system that includes compute resources and shared storage resources, including: receiving, from a data producer, a dataset; storing, within the storage system, the dataset; allocating processing resources to an analytics application; and executing the analytics application on the processing resources, including ingesting the dataset from the storage system.