Metric Data Handling Rules for Multi-Tier Storage Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Centralized metrics management platforms face challenges in managing large volumes of metric data, including increased bandwidth, processing power, and storage requirements, due to the varying and often low value of the data, which can degrade performance and increase costs, and existing solutions like re-programming metric producers are cumbersome and inefficient.
Innovation Solution
A metrics management engine and agent that analyze metric utilization patterns to create intelligent handling rules for storing and transmitting metric data, using multi-tier storage and compression to optimize data handling based on predicted future use.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all metric data is stored in a centralized platform, then complete visibility and analysis capability is achieved, but storage costs and bandwidth requirements increase significantly
Solution Approach 1:
The patent applies local quality by making storage policies metric-specific rather than uniform. Different metrics receive different storage treatments based on their individual characteristics such as usage patterns, importance, and data volume. This allows the system to maintain complete visibility for critical metrics while applying compression or selective storage for less important metrics, thereby reducing overall storage requirements without sacrificing necessary data accessibility.
Solution Approach 2:
The system dynamically changes storage parameters for different metrics based on their characteristics. Metrics are evaluated on parameters such as read frequency, data volume, and business importance, and storage policies are adjusted accordingly. This may include varying compression levels, retention periods, or storage locations for different metrics, optimizing the balance between data visibility and storage cost.
2Quantity of substance
If metric data is compressed, then storage costs are reduced, but data retrieval and processing time increases
Solution Approach 1:
Compression is applied selectively based on metric characteristics rather than uniformly to all metrics. Metrics that are frequently accessed or critical for real-time monitoring may be stored with minimal or no compression, while less critical metrics with larger data volumes undergo higher compression. This localized approach ensures that compression benefits are realized without significantly impacting retrieval performance for important metrics.
Solution Approach 2:
The system applies partial compression actions based on the specific needs of each metric. Instead of fully compressing all data, the system applies compression only where and when it provides net benefit, balancing storage savings against retrieval time requirements for different metric types.
3Loss of substance
If metric producers are re-programmed to filter data, then transmission volume is reduced, but system complexity and maintenance burden increase
Solution Approach 1:
The patent introduces an intermediary component between metric producers and the centralized platform that handles filtering and preprocessing. Rather than modifying metric producers themselves, this intermediary layer applies intelligent filtering rules to reduce transmission volume while keeping the original producers unchanged, thereby avoiding the complexity and maintenance burden of re-programming distributed systems.
Solution Approach 2:
The system enables self-service filtering where the metric management platform automatically determines which metrics to collect and in what format, without requiring configuration changes to metric producers. The platform autonomously manages data filtering, aggregation, and transmission decisions based on centralised policies.
Data Source
AI summary
A computing platform may be configured to (i) receive metric data for a metric that was produced by a metric producer, (ii) identify a metric handling rule that applies to the metric, wherein the identified metric handling rule comprises a handling action of storing metric data for the metric in a specified storage location (e.g., a different tier of a multi-tier storage architecture), and (iii) handle the received metric data for the metric in accordance with the identified metric handling rule by storing the received metric data in the specified storage location.