Storage Management Scheduling for Local and Distributed Data Placement
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage technologies are inflexible and poorly adaptable to different application scenarios, particularly struggling with high read/write operations per second (I/OPS) requirements and throughput needs in big data and desktop cloud environments.
Innovation Solution
A data processing method that divides storage resources into local and distributed storage devices, with a storage management device determining scheduling information to optimize data placement across local and distributed storage pools, allowing for flexible volume types and copy quantities based on data amount and load information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is stored in a single storage location (either local or distributed), then storage simplicity is maintained, but storage flexibility and adaptability to different application scenarios deteriorate
Solution Approach 1:
The patent implements dynamic storage scheduling by allowing the storage management device to flexibly determine whether to store data locally or in the distributed storage pool based on real-time load information and data characteristics. This dynamic allocation mechanism enables the system to adapt to different application scenarios (big data, desktop cloud, etc.) without requiring a fixed storage architecture, thereby resolving the contradiction between storage flexibility and architectural complexity.
Solution Approach 2:
The storage system is segmented into two independent components: local storage devices and distributed storage pool. The storage management device can independently schedule data to either segment based on requirements, allowing flexible combination of local and distributed storage without requiring a completely new unified architecture, thus improving adaptability while controlling complexity.
2Quantity of substance
If all data is stored in distributed storage pool, then storage capacity is improved, but I/OPS performance deteriorates due to network overhead
Solution Approach 1:
The patent applies local quality by allowing frequently accessed data or data requiring high I/OPS performance to be stored locally on the host, while less frequently accessed data is stored in the distributed storage pool. This creates different storage qualities in different locations based on specific data characteristics and access patterns, thereby maintaining high I/OPS performance for critical data while utilizing the capacity of distributed storage for bulk data.
3Speed
If data is cached locally, then read speed is improved, but memory resource consumption increases
Solution Approach 1:
Instead of caching all data locally, the patent implements partial caching by selectively caching only the data that requires high read speed based on access patterns and importance. The storage management device monitors data access and determines which portions of data should be cached locally, applying the partial action principle to balance read speed improvement with memory resource consumption.
Data Source
AI summary
A data processing method to improve data storage flexibility includes receiving, by a first storage management device, a data write request generated by a host, where the host is provided with the first storage management device, determining, by the first storage management device according to the data write request, scheduling information corresponding to the data write request, where the data write request includes to-be-written data, and the scheduling information corresponding to the data write request indicates a distributed storage pool, or a local storage device of the host, and processing, by the first storage management device, the to-be-written data according to the scheduling information corresponding to the data write request.