De-duplicated Storage IO Reduction via Memory Pre-loading
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional de-duplication technologies reduce storage capacity needs but do not decrease the number of input/output (IO) operations required by a server, leading to increased load on the CPU, storage subsystem, and network.
Innovation Solution
Implement a method where a computer tracks and queries a de-duplication engine to identify and store only unique data blocks, allowing memory areas to point to similar blocks, thereby reducing the need for IO operations by serving requests from memory rather than the storage subsystem.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional de-duplication is implemented to reduce storage capacity needs, then storage capacity requirements are reduced, but the number of IO operations remains the same and CPU load increases
Solution Approach 1:
The system pre-loads de-duplicated data blocks into memory before they are actually needed by applications. The de-duplication engine identifies and stores unique data blocks in memory proactively, so when IO requests arrive, the data is already available in memory rather than needing to be retrieved from storage, thus reducing IO operations while maintaining storage efficiency
Solution Approach 2:
The patent introduces memory as an intermediary layer between storage and applications. Instead of applications directly accessing storage for de-duplicated data, the system uses memory as a buffer that holds de-duplicated blocks, mediating between the storage subsystem and application requests, thereby reducing the number of IO operations to storage
2Quantity of substance
If de-duped storage is used to reduce storage capacity, then storage efficiency improves, but CPU load and storage subsystem load increase
Solution Approach 1:
The system performs de-duplication and data block preparation in advance, storing unique blocks in memory before they are needed. This preliminary action shifts the computational workload from CPU-intensive operations during IO requests to batch processing during idle periods, reducing real-time CPU load while maintaining storage efficiency
Solution Approach 2:
The de-duplication engine operates autonomously to identify, track, and store unique data blocks in memory without requiring active CPU intervention during application IO operations. The system self-manages the de-duplication process, reducing the burden on the main CPU and storage subsystem
3Ease of operation
If all data blocks are stored individually in memory, then data accessibility improves, but memory usage increases
Solution Approach 1:
The patent merges multiple identical or duplicate data blocks into a single stored instance in memory. Instead of storing separate copies of identical blocks for different files or requests, the system consolidates them into one shared copy, reducing memory usage while maintaining full accessibility through reference tracking
Solution Approach 2:
The system creates virtual copies or references to data blocks in memory rather than physical duplicates. When the same data block is needed for multiple purposes, the system maintains one physical copy and creates reference pointers, allowing multiple access paths without duplicating the actual data in memory
Data Source
AI summary
The present disclosure relates generally to the field of reducing application input/output operations from a server having data stored on (e.g., remotely stored on) de-duped storage. In various embodiments, reducing application input/output operations from a server having data stored on (e.g., remotely stored on) de-duped storage may be implemented in the form of systems, methods and/or algorithms.