Hierarchical Blacklisting for Distributed Storage Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale distributed storage systems face challenges in managing blacklists efficiently, leading to increased latency and resource wastage due to unresponsive storage components, as existing methods struggle to accurately identify and handle failures in hierarchical system configurations.
Innovation Solution
A system utilizing a blacklist engine that employs hierarchical thresholds to identify and aggregate failed storage elements, excluding them from data distribution and retrieval processes, and dynamically adjusts blacklists based on response status and removal thresholds, ensuring efficient routing of storage operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional blacklisting methods are used in distributed storage systems, then individual failed storage elements can be tracked, but the system continues to attempt operations on storage elements impacted by higher-level failures, resulting in increased latency and resource wastage
Solution Approach 1:
The patent introduces a hierarchical dimension to blacklisting by organizing storage elements into groups and creating blacklists at multiple levels (individual element level, group level, and system level). This hierarchical structure allows the system to efficiently propagate failure information across different granularities, preventing wasted operations on affected storage elements while maintaining fine-grained control where appropriate.
Solution Approach 2:
The patent merges individual storage element blacklists with group-level blacklists to create a comprehensive blacklisting mechanism. By combining these different levels of blacklisting information, the system achieves both precise tracking of individual failures and efficient propagation of higher-level failures, resolving the contradiction between reliability and latency.
2Quantity of substance
If the number of storage elements increases to provide larger storage capacity, then storage capacity and redundancy improve, but the probability of failure of storage elements and controller components increases
Solution Approach 1:
The patent segments the distributed storage system into hierarchical groups and levels, allowing failure management to occur at appropriate granularities. This segmentation enables the system to handle failures more effectively in large-scale configurations by containing failure propagation within groups while maintaining overall system reliability, thus allowing storage capacity to scale without proportionally increasing failure probability.
3Measurement precision
If hierarchical blacklisting with multiple thresholds is implemented, then accurate identification and isolation of unresponsive components improves, but the complexity of blacklist management increases
Solution Approach 1:
The patent uses different threshold parameters at different hierarchical levels to control blacklisting behavior. By adjusting these threshold parameters, the system can achieve precise failure detection and appropriate response levels without requiring complex management logic, as the thresholds provide simple, configurable control mechanisms.
Data Source
AI summary
Example distributed storage systems, controller nodes, and methods provide hierarchical blacklisting of storage system components in response to failed storage requests. Storage elements are accessible through hierarchical storage paths traversing multiple system components. Blacklisted components are aggregated and evaluated against a hierarchy threshold at each level of the hierarchy and all components below the component are blacklisted if the hierarchy threshold is met. Blacklisted components are avoided during subsequent storage requests.