Tiered Storage for Data Processing Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large data sets require significant storage capacity and processing resources, leading to increased costs and complexity, as well as challenges in locating frequently accessed data for optimal processing performance.
Innovation Solution
Implementing tiered storage, where frequently accessed data is stored locally and less frequently accessed data is stored remotely, allowing for dynamic adaptation to changes in access patterns and reducing the need for extensive storage capacity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is stored locally to improve processing performance through locality, then data processing efficiency is improved, but storage capacity requirements increase
Solution Approach 1:
The patent segments data into frequently accessed and less frequently accessed portions, storing the former locally and the latter remotely. This segmentation allows the system to maintain only essential data locally, reducing storage requirements while preserving processing efficiency for critical operations.
Solution Approach 2:
The patent introduces a hierarchical storage dimension with multiple tiers (local and remote storage). This dimensional approach allows data to be accessed from different storage levels based on access frequency, enabling the system to balance between processing speed and storage capacity requirements.
2Quantity of substance
If storage capacity is increased to accommodate large data sets, then more data can be stored and processed, but system complexity and costs increase
Solution Approach 1:
The patent divides the storage system into segmented tiers with different characteristics. By organizing storage capacity into manageable segments (local and remote), the system can scale capacity without proportionally increasing complexity, as each tier can be independently managed and optimized.
3Speed
If all data is stored locally to ensure fast access, then processing performance improves, but resource utilization decreases
Solution Approach 1:
The patent implements a dynamic storage system where data placement between local and remote storage is not static but adapts based on access patterns. This dynamic approach allows the system to optimize resource utilization by moving data between storage tiers as needs change, while maintaining fast access speeds for frequently used data.
Solution Approach 2:
The system uses feedback from data access patterns to dynamically adjust where data is stored. By monitoring access frequency and performance metrics, the system can automatically relocate data between storage tiers, optimizing both access speed and resource utilization based on actual usage patterns.
Data Source
AI summary
Tiered storage may be implemented for processing data. Data processors may maintain some of a data set, including user data and metadata describing the user data, locally. The data set is also maintained a data store remote to the data processor. When processing requests are received, a determination is made as to whether the local portions of the data set can execute the processing request or one or more additional portions of the data set are needed from the remote data store. If additional portions of the data set are needed, then a request may be sent to the data store for the additional portions. Once received, the data processor may execute the processing request utilizing the additional portions. Portions of the data set maintained locally at the data processor may be selected and flushed from local storage to the remote data store.