Hot-Cold Layered Storage for Time Series Query Speed
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Balancing the need for quick data access with minimal delay and reducing storage costs is challenging for large volumes of time series data, particularly in systems like IoT networks.
Innovation Solution
Implementing a layered data storage approach with a hot storage layer for fast query access and a cold storage layer for cost-effective data retention, using mechanisms to expire older data and manage data between layers through compaction and deduplication techniques.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored in expensive fast storage infrastructure, then query response latency is reduced, but storage costs increase
Solution Approach 1:
The patent segments storage infrastructure into hot storage layer (fast, expensive) and cold storage layer (slow, inexpensive). Recent frequently accessed data is stored in hot storage for low latency queries, while older data is moved to cold storage to reduce costs. This segmentation resolves the contradiction by matching storage speed to actual data access patterns.
Solution Approach 2:
The patent applies local quality by providing different storage characteristics to different data based on its age and access patterns. Recent data receives high-speed storage resources, while older data is relegated to lower-cost storage. This ensures that only the data requiring fast access consumes expensive storage resources.
2Speed
If data is retained in hot storage layer, then query access speed is maintained, but storage costs increase
Solution Approach 1:
The patent implements dynamic data movement between hot and cold storage layers based on data age and access patterns. The system continuously monitors and relocates data, ensuring that hot storage contains only the subset of data requiring fast access, while older data is automatically migrated to cold storage.
3Quantity of substance
If data is compressed and stored in columnar format, then storage efficiency increases, but query complexity increases
Solution Approach 1:
The patent segments query processing into two paths: simple recent data queries access hot storage in record-oriented format for ease of access, while complex historical queries access cold storage in columnar format optimized for analytics. This segmentation allows each storage layer to be optimized for its primary use case without compromising overall system performance.
Data Source
AI summary
This disclosure relates to, among other things, scalable data processing, storage, and/or management systems and methods. In various embodiments, sequence number schemes associated with ingested data records may allow for improved management of data stored in one or more layers (e.g., hot and/or cold data storage layers) using data merging, deletion, and/or compaction techniques. In certain embodiments, certain data records stored in one or more segment files associated with a partition may be merged into compacted segment files, thereby reducing storage overhead associated with storing prior records that have been updated and/or deleted records. In further embodiments, data record tombstones may be employed in connection with the management of deleted records.