Storage Telemetry Data Reduction via Algorithm Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Storage arrays face limitations in transmitting telemetry data due to bandwidth constraints, leading to reduced data availability and usefulness, especially when only a subset of data is transmitted to conserve bandwidth, which introduces delays and limits information extraction.
Innovation Solution
Implementing a data reduction algorithm selection process that includes de-duplication and compression, matching current telemetry data with historical data to determine the most effective algorithm for reducing data transmission while ensuring it runs at a rate comparable to data generation, and removing unnecessary data if the selected algorithm does not sufficiently reduce bandwidth usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If telemetry data is transmitted over bandwidth-limited communication connections, then data transmission is possible, but the amount of useful information that can be extracted is limited
Solution Approach 1:
The patent extracts and transmits only the most critical telemetry data elements over bandwidth-limited connections. By selectively removing non-essential data points and focusing on key performance indicators, the system maintains data usefulness while adapting to communication constraints.
Solution Approach 2:
The patent applies different data transmission quality levels to different telemetry data types. Critical data receives higher fidelity transmission while less important data is aggregated or summarized, optimizing the balance between data volume and information value across the bandwidth-limited channel.
2Quantity of substance
If telemetry data transmission bandwidth is increased, then more complete data can be sent, but customer site bandwidth is consumed
Solution Approach 1:
The patent extracts only the essential telemetry data elements needed for monitoring and analysis, excluding redundant or less critical information. This extraction approach reduces the total data volume transmitted, thereby conserving customer site bandwidth while maintaining data effectiveness.
Solution Approach 2:
The patent implements partial data transmission by sending a curated subset of telemetry data rather than complete datasets. This partial action approach provides sufficient information for operational monitoring without consuming excessive customer bandwidth resources.
3Quantity of substance
If data reduction algorithms are applied to telemetry data, then transmission bandwidth is reduced, but processing time increases
Solution Approach 1:
The patent applies data reduction algorithms as preliminary processing steps before data transmission. By pre-compressing and pre-filtering telemetry data at the source, the system reduces the burden on transmission channels and minimizes real-time processing requirements during data reception and analysis.
Solution Approach 2:
The patent dynamically adjusts data reduction parameters based on available bandwidth, data criticality, and processing capacity. By changing compression ratios, sampling rates, and aggregation levels adaptively, the system optimizes the balance between reduced transmission size and acceptable processing time.
Data Source
AI summary
Reducing telemetry data transmitted from a storage system includes selecting a data reduction algorithm from a plurality of data reduction algorithms, where selection includes eliminating data reduction algorithms that process frames of telemetry data at a lower rate than the frames of telemetry data are generated and applying the data reduction algorithm to the telemetry data. Selecting a data reduction algorithm may include selecting a remaining one of the data reduction algorithms that provides a greatest amount of data reduction. Reducing telemetry data transmitted from a storage system may also include determining if the data reduction algorithm sufficiently reduces an amount of data that is transmitted using a communications channel and removing some of the telemetry data in response to the data reduction algorithm not sufficiently reducing the amount of data that is transmitted. The data reduction algorithms may include data de-duplication and data compression.