Disaggregated Storage IO Distribution via Flash-Aware Weighted Mapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage systems fail to efficiently manage flash devices at scale, particularly low endurance flash storage like QLC SSDs, and struggle with asymmetric scaling and synchronization in e-commerce workloads, leading to challenges in resource management and heterogeneous storage control.
Innovation Solution
A flash-aware distributed storage system that statically maps IO metadata and data to controllers based on IO management weights, determined by controller parameters and network type, allowing for efficient, flexible, and scalable management of flash devices without the need for synchronization mechanisms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If data and metadata are statically mapped to storage nodes in share nothing paradigm, then storage node management is simplified, but flash device endurance and efficiency are degraded
Solution Approach 1:
The system segments the storage architecture into compute nodes and storage nodes with distinct responsibilities. Compute nodes handle IO request processing and metadata management, while storage nodes专注于数据存储和闪存的物理管理。这种 segmentation 使得每个节点可以优化其功能，计算节点可以实施复杂的闪存感知算法，而存储节点保持简单高效
Solution Approach 2:
引入计算节点作为中介层，在客户端和存储节点之间建立缓冲。计算节点负责生成IO元数据、确定控制器参数、计算优先级权重和IO管理权重，然后才能将数据映射到存储节点。这种中介机制避免了存储节点的复杂决策，同时实现了闪存感知的智能调度
2Adaptability or versatility
If data and metadata are mapped to any storage node in share everything paradigm, then resource utilization is improved, but synchronization complexity and network traffic increase
Solution Approach 1:
将元数据管理功能从存储节点中提取出来，集中到计算节点。所有IO元数据都存储在计算节点，存储节点只保存纯数据。这样消除了存储节点之间的元数据同步需求，因为每个存储节点的数据在逻辑上由特定计算节点管理，避免了共享Everything范式中的同步复杂性问题
Solution Approach 2:
在数据写入存储节点之前，计算节点预先完成所有决策：确定目标控制器、计算权重、生成元数据。这种预先行动确保了数据分布的优化，同时避免了后续的同步需求，因为映射关系在写入前就已经确定
3Device complexity
If conventional storage systems are used for flash devices, then system simplicity is maintained, but flash device efficiency and endurance management are degraded
Solution Approach 1:
系统在局部层面实现了闪存感知优化。每个计算节点根据其管理的存储节点和控制器的具体特性（如闪存类型QLC/PLC、控制器参数、网络类型RDMA/TCP）动态计算权重和映射策略。这种局部优化使得每个闪存设备都能根据其特性得到最佳管理，而不是采用一刀切的常规存储方法
Solution Approach 2:
系统通过动态改变多个参数来实现闪存高效管理：根据控制器参数（CPU、RAM、NIC容量）确定优先级权重，根据网络类型（RDMA/TCP）调整权重分配，根据闪存类型（QLC/PLC）优化数据分布。这些参数的灵活调整使得系统能够适应不同闪存设备的特性，大幅提升效率
Data Source
AI summary
A method for distributing and managing an Input/Output (IO) request in a disaggregated storage architecture includes receiving the IO request including IO data to be distributed in the disaggregated storage architecture, generating IO metadata corresponding to the IO data included in the received IO request, determining one or more controller parameters for each of a plurality of controllers of the disaggregated storage architecture, determining a first priority weight of each controller parameter of the one or more controller parameters based on a network type of the disaggregated storage architecture, determining a first IO management weight for each of the plurality of controllers based on the one or more controller parameters and corresponding first priority weights, and statically mapping each of the IO metadata and the IO data to at least one controller of the plurality of controllers based on the first IO management weights of the plurality of controller.