Storage System Data Placement for Cost and Speed Trade-offs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Content addressable storage systems face high costs for storing large amounts of data, and existing solutions do not efficiently manage storage between expensive and less expensive storage devices, leading to inefficient use of resources.
Innovation Solution
Implementing a system with a primary storage system using more expensive devices and a secondary storage system using less expensive devices, where content is replicated from the primary to the secondary system based on user-defined policies, allowing for deletion from the primary system to free up space and maintain access through the secondary system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored on expensive high-performance storage devices, then data accessibility and speed are improved, but storage costs increase
Solution Approach 1:
The storage system is segmented into multiple storage devices with different performance characteristics (fast/expensive and slow/cheap). Data is divided and distributed across these segments based on access patterns, allowing frequently accessed data to reside on fast storage while less frequently accessed data moves to slow storage, resolving the contradiction between speed and cost.
Solution Approach 2:
The system dynamically moves data between fast and slow storage devices based on changing access patterns and policies. Data that was previously accessed frequently may be moved to slow storage after a certain period, while recently accessed data is moved back to fast storage, allowing the system to adapt to varying workloads and optimize the balance between speed and cost over time.
2Quantity of substance
If data is moved to less expensive storage devices, then storage costs are reduced, but data access speed decreases
Solution Approach 1:
The system performs preliminary actions by predicting which data will be needed in the future and pre-loading it into fast storage before it is actually requested. This anticipatory approach ensures that when data is accessed, it is already in the fast storage layer, maintaining high access speeds while allowing the system to keep slower storage devices utilized for cost efficiency.
Solution Approach 2:
The system dynamically adjusts the distribution of data between fast and slow storage based on real-time and historical access patterns. By continuously monitoring and adapting data placement, the system ensures that data access speed requirements are met while minimizing storage costs through efficient use of slower, cheaper storage resources.
3Reliability
If data is replicated from primary to secondary storage system, then data availability is improved, but storage resource usage increases
Solution Approach 1:
The system creates copies of data and places them in different storage locations (primary and secondary storage systems). This copying approach ensures data availability and reliability by having redundant copies, while the system intelligently manages the replication process to optimize storage resource usage based on policies and access patterns.
Solution Approach 2:
Different qualities of storage are assigned to different data based on local requirements. Critical data that requires high availability is replicated across multiple systems, while less critical data may be stored only in the primary system. This differentiated approach optimizes both reliability and storage resource usage by applying replication selectively rather than uniformly.
Data Source
AI summary
One embodiment is directed to a computer system that includes primary and secondary storage systems. When a request to store a content unit is received, it may be determined whether to store the content unit on the primary storage system or the secondary storage system. In another embodiment, a computer may store information relating to an action to be performed in one of the directories of a file system that corresponds to a period of time. When the period of time to which the directory corresponds arrives, the action may be performed. In another embodiment, a content unit stored on a primary storage system may be copied to a secondary storage system in a computer system. After the content unit has been copied, the secondary storage system may send a delete request to the primary storage system to delete the content unit from the primary storage system.