Heterogeneous Storage Cluster Scheduling for Low-Latency Data Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed virtual storage systems face challenges in improving data access performance, reducing latency, and minimizing network bandwidth and storage resource consumption due to the use of different access protocols and interfaces across heterogeneous storage systems.
Innovation Solution
A method and system utilizing a non-dominated sorting genetic algorithm to construct a multi-objective optimization model for selecting optimal storage positions, minimizing total storage cost, maximizing throughput, and minimizing distance between datasets and storage systems, thereby optimizing data access and resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional local file system storage mechanism is used, then data is persistently stored on hard disk, but data access performance is poor and latency is high
Solution Approach 1:
The patent pre-loads frequently accessed data from hard disk storage into memory before actual access requests occur. The system predicts which data will be needed and loads it into the memory buffer in advance, so when access is requested, the data is already available in fast memory, significantly reducing access latency while maintaining persistent storage on disk
Solution Approach 2:
The patent introduces a memory buffer as an intermediary layer between the hard disk storage and the access requests. This buffer cache acts as a mediator that holds copies of frequently accessed data, allowing the system to serve requests from fast memory rather than slow disk, thereby improving access performance without sacrificing persistent storage reliability
2Speed
If data is loaded into memory for processing, then data access speed improves, but network bandwidth and storage resources are unnecessarily consumed
Solution Approach 1:
The patent applies local quality by making the memory buffer size and data retention policy specific to each data access pattern and frequency. Instead of uniformly loading all data into memory, the system dynamically adjusts which data resides in memory based on local access characteristics, ensuring that only necessary data occupies memory resources, thereby improving access speed for critical data while avoiding unnecessary network bandwidth consumption for less frequently accessed data
3Productivity
If multiple data centers concurrently process massive data, then data processing capacity increases, but data migration or duplicate storage increases resource consumption
Solution Approach 1:
The patent creates a unified virtual storage namespace that can be accessed by multiple data centers simultaneously. Instead of requiring separate copies of data in each data center, the system provides a universal access interface that allows concurrent read/write operations on the same data across multiple locations, enabling increased processing capacity without the need for duplicate storage or extensive data migration
Data Source
AI summary
The present disclosure provides a method and a system for managing and scheduling a cross-domain heterogeneous storage cluster. With optimization objectives of minimizing the total storage cost, maximizing the total throughput and minimizing a distance between a to-be-stored dataset and a position of a storage system where the storage is about to perform, constraint conditions are solved by adopting a non-dominated sorting genetic algorithm, and an optimal storage position is selected. Accordingly, the data access performance is improved, the latency is reduced, and the unnecessary consumption of network bandwidth and storage resources is reduced.