Ingest-Time Data Reduction for Storage Bandwidth Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face limitations in ingest bandwidth due to the bottleneck caused by non-volatile random access memory (NVRAM) devices, as deduplication and compression are performed only after host data is placed in the logger, restricting the ability to ingest data from a host computer effectively.
Innovation Solution
Performing data reduction operations such as deduplication and compression on raw host data in the data cache during ingest, allowing reduced data to be ingested, which can exceed the connecting bandwidth limitations of NVRAM devices, and optionally writing reduced data directly into secondary storage to bypass NVRAM altogether, with the option to enable/disable this process based on I/O operations or bandwidth conditions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If deduplication and compression are performed after host data is placed in the logger, then data reduction is achieved, but the ingest bandwidth is limited by the connecting bandwidth of the NVRAM devices
Solution Approach 1:
The patent performs data reduction operations (deduplication and compression) on host data during the ingest process, before the data is placed in the logger. This preliminary action reduces the volume of data that needs to be transferred to and stored in the NVRAM-based logger, thereby preventing the logger's connecting bandwidth from becoming a bottleneck to the overall ingest bandwidth
Solution Approach 2:
The patent extracts the data reduction operations from the traditional post-logger position and moves them to occur during the ingest process itself. By taking out the data reduction step and performing it earlier in the data flow, the system reduces the amount of data that must pass through the limited-bandwidth logger connection
2Productivity
If additional NVRAM devices are provisioned to increase connecting bandwidth to the logger, then ingest bandwidth is improved, but the cost of the data storage system increases
Solution Approach 1:
The patent changes the parameter of data volume by performing reduction operations during ingest. Instead of increasing hardware capacity (NVRAM devices) to handle full-volume data, the system transforms the data to occupy less space, thereby achieving higher effective ingest bandwidth with the same hardware configuration and avoiding additional costs
3Productivity
If data reduction operations are performed during ingest, then ingest bandwidth exceeds logger connecting bandwidth limitations, but the complexity of the data storage system increases
Solution Approach 1:
The patent implements a multi-functional architecture where the data storage system can operate in multiple modes: traditional mode (without ingest-time reduction) and enhanced mode (with ingest-time reduction). The system universally handles both scenarios, allowing it to adapt to different workload requirements and logger bandwidth conditions without requiring separate dedicated systems
Solution Approach 2:
The patent introduces dynamic control mechanisms that allow the data storage system to enable or disable ingest-time data reduction based on real-time conditions such as logger bandwidth utilization and workload characteristics. This dynamic adjustment optimizes performance without permanently increasing system complexity
Data Source
AI summary
A technique performs data reduction on host data of a write request during ingest under certain circumstances. Therein, raw host data of a write request is placed from the host into a data cache. Further, a data reducing ingest operation is performed that reduces the raw host data from the data cache into reduced host data (e.g., via deduplication, compression, combinations thereof, etc.). After completion of the data reducing ingest operation, a late-binding operation is performed that updates a mapper with ability to access the reduced host data from secondary storage. Such ingest-time data reduction may be enabled/disabled (e.g., turned on or off) per input/output (I/O) operation (e.g., used only for relatively large asynchronous I/O operations) and/or activated in situations in which the ingest bandwidth is becoming a bottleneck.