Tiered Queuing System for Data Storage Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing queuing systems face challenges in efficiently managing large volumes of data across multiple regions, leading to issues with data access, storage costs, and fault tolerance, particularly when using cold storage for long-term data retention.
Innovation Solution
A tiered queuing system is implemented, which includes a method for identifying segments to be evicted from local storage and moving them to cold storage, using a cloud service provider for storage and enabling automatic retrieval of data in response to read requests, regardless of the region.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored in local storage across multiple regions, then data access speed is improved, but storage costs and network bandwidth utilization increase
Solution Approach 1:
The patent segments data into hot data (frequently accessed) and cold data (infrequently accessed), storing hot data in local storage across multiple regions for fast access, and cold data in a single region's local storage to reduce network bandwidth utilization and storage costs
Solution Approach 2:
The patent introduces a tiered storage architecture with two levels: local storage for hot data and cloud storage for cold data. This dimensional change in storage hierarchy allows the system to optimize for both speed (local storage) and cost-efficiency (cloud storage) simultaneously
2Reliability
If data is replicated across multiple regions for fault tolerance, then reliability is improved, but network saturation and bandwidth consumption increase
Solution Approach 1:
The patent applies local quality by replicating only the necessary portions of data (hot data) to multiple regions for fault tolerance, while keeping cold data in a single region. This selective replication reduces network bandwidth consumption while maintaining adequate reliability
3Loss of energy
If cold storage is used for long-term data retention, then storage costs are reduced, but data access latency increases
Solution Approach 1:
The patent implements preliminary action by pre-loading cold data into local storage in the same region before access is needed. This anticipatory loading reduces access latency when cold data is required, while still maintaining the cost benefits of cold storage for long-term retention
4Reliability
If data is stored in cold storage in a different region, then disaster recovery capability is improved, but data copying bandwidth and time increase
Solution Approach 1:
The patent applies partial action by replicating only critical hot data to a different region for disaster recovery, rather than copying all data. This selective replication reduces data copying bandwidth consumption while maintaining disaster recovery capability for the most important data
Data Source
AI summary
Disclosed herein are methods, apparatuses, and systems for operating a tiered queuing system that includes: identifying a first segment to be evicted from a local storage in a first region based on an eviction policy; storing the first segment in a cold storage provided by a cloud service provider in the first region; deleting at least one instance of the first segment from the local storage in the first region; indicating in a segment index that the first segment is stored in the cold storage; responsive to receipt of a read request, reading an entry in the segment index for a second segment; and responsive to an indication that the second segment is stored by the cloud service provider, copying the second segment to the local storage and responding to the read request using the second segment stored in the local storage.