Multi-Cloud Storage System for Low Latency Data Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage systems face challenges in providing low latency and efficient access to large volumes of data, especially in high-performance environments like data centers, where advanced storage functions such as snapshots and meta data tracking are desired.
Innovation Solution
A storage system architecture that utilizes multiple cloud services, including a compute engine cloud service for handling user requests, a database cloud service for storing file and folder representations, a high performance object storage service for meta data and snapshot versioning, and a lower performance object storage service for storing individual stripes, to achieve optimized performance and resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If a single high-performance storage service is used, then access speed is improved, but resource utilization and cost efficiency deteriorate
Solution Approach 1:
The storage system is segmented into multiple cloud services with different performance characteristics. Hot data requiring fast access is stored in high-performance storage services, while cold data is stored in lower-performance, more cost-effective services. This segmentation allows the system to optimize for both speed and resource utilization by matching data access patterns with appropriate storage performance levels.
2Loss of energy
If multiple cloud services are integrated, then resource utilization is optimized, but system complexity increases
Solution Approach 1:
A file system interface acts as an intermediary layer between users and the multiple cloud services. This intermediary abstracts the complexity of managing multiple storage services with different performance characteristics, presenting a unified interface to users while handling the complex routing and data placement logic internally. This resolves the contradiction by hiding system complexity from users while maintaining optimized resource utilization.
3Loss of time
If fast storage services are used for all data, then access latency is reduced, but storage cost increases
Solution Approach 1:
Different quality levels of storage services are assigned to different data based on local requirements. Frequently accessed data (hot data) is placed in fast storage services to minimize access latency, while infrequently accessed data (cold data) is placed in slower, cheaper storage services. This local quality differentiation resolves the contradiction by applying fast storage only where necessary, reducing overall storage costs while maintaining low latency for critical operations.
Data Source
Figure 1
Figure 2
Figure 3a
AI summary
A method for implementing a storage system is described. The method includes accepting a filepath from a user that specifies a file. The method includes forming a primary key for a representation of the file. The method includes applying the primary key to a database cloud service to obtain a representation of the file. The representation of the file includes an inode for the files meta data. The method includes using the inode for the files meta data to obtain the files meta data from a high performance object cloud storage service. The files meta data points to information within the high performance object cloud storage service for accessing the files stripes. The method includes accessing the information within the high performance object cloud storage service to obtain an object ID for a stripe within the file.