Clustered Storage Routing with Weighted Retrieval and Local Caching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Legacy distributed data storage clustering solutions face inefficiencies in data placement and request routing, leading to increased latencies and suboptimal retrieval performance.
Innovation Solution
A computer-implemented method that utilizes a record management system to manage data storage and retrieval across clustered storage frameworks by applying predefined placement rules, determining cluster weights based on performance metrics, and caching frequently accessed data in high-performance storage media to reduce retrieval latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored across multiple storage clusters for redundancy and scalability, then data reliability and system capacity are improved, but data retrieval latency increases due to cluster selection challenges
Solution Approach 1:
The system performs preliminary actions by pre-calculating and storing cluster weights and rankings in a storage medium before data retrieval operations. When a data retrieval request arrives, the system queries the storage medium to obtain pre-computed cluster information, eliminating the need for real-time cluster evaluation and significantly reducing retrieval latency while maintaining reliability through multi-cluster storage
Solution Approach 2:
The system implements feedback mechanisms by continuously monitoring data retrieval performance and using this information to dynamically update cluster weights. The storage manager receives performance data, processes it to adjust cluster rankings, and stores updated weight information in the storage medium. This feedback loop enables the system to optimize future retrieval operations based on actual performance patterns
2Measurement precision
If the system queries storage medium for cluster information during data retrieval, then retrieval accuracy is improved, but processing time increases
Solution Approach 1:
The system performs preliminary action by pre-querying and storing cluster information in the storage medium during system initialization or periodic updates. When data retrieval is needed, the system simply reads from the pre-stored information rather than performing new queries, maintaining high retrieval accuracy while minimizing processing time
Solution Approach 2:
The system creates a copy of cluster information in the storage medium that can be rapidly accessed during data retrieval operations. This copied information includes cluster weights, rankings, and metadata, allowing the system to make informed retrieval decisions without performing real-time queries to external sources
3Productivity
If the storage system implements weighted cluster selection based on performance metrics, then data retrieval efficiency is improved, but system complexity increases
Solution Approach 1:
The system extracts the complexity of performance monitoring and cluster evaluation by separating it into distinct components: a storage manager that handles performance data processing, a storage medium that stores cluster information, and retrieval operations that simply query and use the stored data. This extraction reduces the complexity burden on individual retrieval operations
Solution Approach 2:
The storage medium acts as an intermediary between the complex performance monitoring system and the simple data retrieval operations. It buffers the complexity by storing pre-processed cluster information, allowing retrieval operations to remain simple while the system maintains sophisticated performance-based selection through the intermediary layer
Data Source
AI summary
This disclosure describes techniques for enabling efficient and/or expeditious retrieval of data records stored across a clustered storage framework. An example system manages storage of data records in the storage clusters based on predefined placement rules. Frequently accessed data records are stored in a local high-performance store storage medium to reduce retrieval latency. The system determines which storage cluster to retrieve a requested data record from by selecting the cluster with the highest weight, as determined based on performance-related metrics like cluster utilization and geographic proximity.