Intelligent Object Replication for Container Storage Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud storage facilities impose access constraints on buckets, leading to delays and increased costs due to limitations in the number of concurrent accesses, especially in large-scale environments where many containers share common layers.
Innovation Solution
A container image storage system implements intelligent replication by determining the number of object copies based on a reference count across different container images, using a logarithmic function to balance storage usage and ensure efficient concurrent access, thereby reducing bottlenecks and costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If a single copy of an object is maintained in the object store to reduce storage usage, then storage costs are reduced, but concurrent access performance deteriorates due to bottlenecks
Solution Approach 1:
The system dynamically determines the number of object copies to maintain based on real-time reference counts and access patterns. The replication factor is not fixed but adapts to the actual usage conditions, allowing the system to optimize between storage efficiency and access performance for each object individually
Solution Approach 2:
Different objects receive different replication strategies based on their individual characteristics. Objects with high reference counts and concurrent access requirements receive more copies, while objects with low reference counts maintain minimal replication. This local optimization resolves the contradiction by applying the right replication level to each object rather than using a uniform approach
2Productivity
If multiple copies of an object are created to facilitate concurrent accesses, then access performance is improved, but storage usage increases
Solution Approach 1:
The system continuously monitors reference counts and access patterns to feedback-adjust the number of object copies. When reference counts increase indicating higher demand, the system automatically increases replication. When reference counts decrease, the system reduces replication to save storage space, creating a closed-loop optimization system
Solution Approach 2:
The system changes the replication parameter (number of copies) based on varying reference counts and access patterns. By adjusting this parameter dynamically rather than using a fixed value, the system optimizes the balance between storage consumption and access performance for each object according to its actual usage characteristics
3Quantity of substance
If access constraints are imposed on storage buckets to manage costs, then storage costs are controlled, but access delays increase
Solution Approach 1:
The system segments the object store into multiple independent copies of the same object distributed across different storage locations. This segmentation allows concurrent access requests to be distributed across multiple copies rather than competing for a single resource, effectively managing access constraints while minimizing delays
Solution Approach 2:
The system adds the dimension of replication factor to the traditional single-copy storage model. By operating in a multi-dimensional space where objects can exist as multiple copies with different replication factors, the system creates additional access pathways that bypass the limitations of single-bucket access constraints
Data Source
AI summary
Scalable object storage with intelligent replication is disclosed. A container image storage system executing on one or more processor devices receives a container image comprising a plurality of objects. For each object, a reference count indicative of how many different container images stored in the container image storage system include the object is determined. For each object, a number of copies of the object to be stored in a storage based on the reference count is determined. For each object, the number of copies of the object are stored in the storage.