Adaptive Data Retention Models for IoT Edge Storage Constraints
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing techniques for storing and retaining data in resource-constrained environments, such as IoT edge gateway devices, are inadequate for managing the varying criticality and importance of data from numerous IoT devices, leading to inefficient use of storage resources.
Innovation Solution
Adaptive storage resource usage techniques that involve sampling data, fitting data models to obtain representations, and classifying data into predefined retention models to optimize storage resource usage, including varying data retention models, evicting data, moving data to different storage tiers, and determining storage time based on data age and criticality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored in caches and storage devices in resource-constrained environments, then data retention capability is improved, but storage resource efficiency deteriorates
Solution Approach 1:
The patent applies local quality by differentiating data storage strategies based on data criticality and importance. Different retention models (complete retention, subsample retention, lossy retention) are applied to different data sources or data types, allowing optimal storage resource allocation where high-criticality data receives comprehensive retention while low-criticality data uses more efficient retention strategies.
Solution Approach 2:
The system dynamically adapts storage resource usage by continuously monitoring data characteristics and adjusting retention policies accordingly. The adaptive mechanism allows the system to shift between different retention models based on changing data patterns, ensuring optimal balance between data retention capability and storage resource efficiency over time.
2Reliability
If comprehensive data retention is implemented, then data availability is improved, but storage cost increases
Solution Approach 1:
Instead of applying uniform comprehensive retention to all data, the system applies local quality by tailoring retention strategies to specific data sources based on their criticality. High-criticality data sources receive complete retention for maximum availability, while low-criticality sources use subsample or lossy retention models that reduce storage costs while maintaining adequate availability for analytical purposes.
Solution Approach 2:
The system changes retention parameters (retention period, sampling frequency, data granularity) based on data source characteristics and criticality assessments. By adjusting these parameters dynamically, the system optimizes the balance between data availability and storage cost, retaining enough data to meet analytical needs while avoiding excessive storage expenditure on low-value data.
3Measurement precision
If data models are fitted to sampled data, then data representation accuracy is improved, but processing complexity increases
Solution Approach 1:
The system applies partial action by fitting data models to sampled data rather than processing all raw data. This sampling approach maintains sufficient representation accuracy for analytical purposes while significantly reducing processing complexity and computational resource requirements compared to full-data processing.
Solution Approach 2:
The system creates simplified representations (copies) of data through fitted models that capture essential patterns and characteristics. These model representations serve as substitutes for the full raw data in analytical operations, maintaining measurement precision while reducing the complexity of subsequent processing tasks.
Data Source
AI summary
Techniques are provided for adaptive usage of storage resources using data source models and data source representations generated using the data source models. One method comprises obtaining sampled data generated by sampling source data from a data source; fitting a data model to the sampled data to obtain a representation of the sampled data; obtaining a classification of the sampled data into one of multiple predefined retention models; and adapting a usage of one or more storage resources that store the retained data based on the representation and the classification. The adaptive storage resource usage may comprise, for example: (i) varying a data retention model based on an age of the sampled data; (ii) evicting cache data based on the representation; (iii) moving the retained data to a different storage tier; and (iv) determining an amount of time to store the retained data.