Erasure-Coded Data Storage Node Scaling for Hot Data Reads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed data storage systems face challenges in optimizing read times and resource utilization, as they require expensive faster hardware for hot data and careful management to adapt to changing data access patterns, leading to inefficiencies in resource allocation and performance.
Innovation Solution
A computer-implemented system that monitors data object utilization and availability, using erasure coding to dynamically adjust the number of data storage nodes based on usage patterns, allowing for increased or decreased storage nodes to optimize read times and resource usage by modifying the number of data blocks and nodes storing coded data objects.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If faster hardware is used to store hot data, then read time is improved, but system cost increases
Solution Approach 1:
The patent implements dynamic adjustment of the number of storage nodes based on real-time data access patterns. When data is frequently accessed (hot data), more nodes are allocated to improve read speed. When access frequency decreases, nodes are reduced to lower costs. This dynamic adaptation resolves the contradiction by making the system flexible rather than static.
Solution Approach 2:
The system changes the parameter of storage node quantity based on data utilization metrics. By monitoring access patterns and adjusting the number of nodes accordingly, the system optimizes the balance between read performance and resource costs, avoiding the need for consistently expensive high-performance hardware.
2Speed
If more fast hardware is added to handle increasing hot data volume, then read performance is maintained, but hardware management complexity increases
Solution Approach 1:
The system performs self-adjustment by automatically monitoring data access patterns and dynamically modifying the number of storage nodes without manual intervention. This self-service capability eliminates the need for complex manual hardware management while maintaining optimal read performance.
Solution Approach 2:
The system implements a feedback loop where data utilization metrics are continuously monitored and fed back to the control logic, which then adjusts the number of storage nodes accordingly. This automated feedback mechanism simplifies hardware management by replacing complex manual processes with intelligent self-regulation.
3Ease of operation
If data is statically allocated to fixed hardware tiers, then hardware management is simplified, but adaptability to changing access patterns decreases
Solution Approach 1:
The patent transforms the static hardware tiering model into a dynamic one where the number of storage nodes is continuously adjusted based on actual data access patterns. This dynamic approach maintains ease of operation through automated management while significantly improving adaptability to changing workloads.
Data Source
AI summary
In part, the disclosure relates to a computer-implemented system for managing a data storage system comprising a plurality of data storage nodes. The system includes at least one processor and an operatively associated storage device. The at least one processor is programmed to: monitor utilization of at least one data object stored at the data storage system as a coded data object comprising a first total number of data blocks, wherein each of the first number of data blocks is stored at a separate data storage node, and wherein an original data object is re-creatable from a first base number of data blocks selected from the first total number of data blocks; determine that the utilization of the at least one data object has changed; and modify a number of the plurality of data storage nodes that store data blocks of the coded data object.