Dynamic IoT Data Retention Using Query And Load Statistics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
IoT platforms face challenges in managing data retention policies due to varying data relevance and query requirements, leading to storage capacity issues, operational inefficiencies, and the need for manual adjustments that can result in suboptimal performance and high costs.
Innovation Solution
A dynamic data retention framework that automatically adapts retention policies based on load and query statistics, ensuring relevant data remains in the operational database while less relevant data is deleted, leveraging both operational and data lake storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data retention policies are manually defined and adjusted, then data availability requirements can be met, but operational costs increase and performance may be suboptimal
Solution Approach 1:
The system automatically monitors query patterns and data access frequencies, then dynamically adjusts retention policies without human intervention. The platform self-optimizes by identifying frequently accessed data and extending its retention period automatically, eliminating manual operations while maintaining data availability requirements.
Solution Approach 2:
The system continuously monitors query statistics and data access patterns, using this feedback to dynamically adjust retention policies. When query frequency for certain data increases, the system automatically extends retention; when access decreases, it reduces retention periods, creating a closed-loop system that optimizes both availability and costs.
2Reliability
If all incoming data is stored forever in an internal database, then complete data availability is achieved, but storage requirements and platform load increase significantly
Solution Approach 1:
The system implements dynamic retention policies where the retention period for each data set is adjusted based on actual query patterns and access frequencies. Instead of a static 'store forever' approach, the system continuously adapts retention durations to match actual data value and demand, optimizing storage utilization while maintaining availability of relevant data.
Solution Approach 2:
The system applies different retention policies to different data sets based on their individual access patterns and importance. Frequently accessed operational data receives extended retention, while rarely accessed historical data has shorter retention periods, creating a differentiated storage strategy that optimizes overall system performance and storage efficiency.
3Reliability
If manual data retention policies are used, then initial data requirements can be met, but the system cannot adapt to changing use cases and device connections
Solution Approach 1:
The system continuously monitors changes in query patterns, device connections, and data access frequencies, dynamically adjusting retention policies in response to evolving requirements. When new use cases emerge or device patterns change, the system automatically adapts retention periods to maintain optimal data availability without manual reconfiguration.
Solution Approach 2:
The system uses real-time feedback from query statistics and access patterns to automatically adjust retention policies as use cases evolve. This closed-loop approach ensures the system adapts to changing requirements by continuously learning from actual data usage and modifying retention strategies accordingly.
4Quantity of substance
If data is offloaded to a data lake for long-term storage, then storage costs are reduced, but query response times increase due to slower access
Solution Approach 1:
The system maintains frequently accessed hot data in the high-performance operational database while offloading cold, rarely accessed data to the cost-effective data lake. This differentiated storage strategy ensures that data requiring fast access remains in fast storage, while data suitable for archival moves to cheaper storage, optimizing both performance and costs.
Solution Approach 2:
The system dynamically determines which data resides in the operational database versus the data lake based on real-time access patterns and query requirements. Frequently accessed data is kept in the operational database for fast retrieval, while less frequently accessed data is stored in the data lake, with automatic adjustments as access patterns change.
Data Source
AI summary
Data storage management is carried out by an Internet of Things platform having access to operational and long-term databases carrying tables for storing device-related data emitted from connected devices. Device-related data is received and stored in operational database table entries. Tables are associated with devices. Table entries are associated with entry dates. A table in the operational database is monitored to obtain load statistics. Data from the operational database is offloaded by replication to the long-term database. Queries against operational database tables are tracked to obtain query statistics. Cleanup rules determining timeframes within which data stored in operational database table entries are to be deleted are set up, based on a date of data entry. Cleanup rules currently set up are carried out as a retention run, based on the load statistics and/or query statistics.