Container Image Storage Filtering for Network Load Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face inefficiencies and resource bottlenecks when handling heavy workload requests for data access, leading to network and storage system overload, as well as challenges in accurately measuring and monetizing resource utilization.
Innovation Solution
Implementing a container image on the storage system to manage workload requests, allowing for localized data processing, resource cost association, and strategic allocation between primary and secondary storage, with features like compute summaries and consumption limits to optimize resource usage and data transfer.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is accessed directly from storage system over network, then data availability is improved, but network traffic and storage system load increase causing bottlenecks
Solution Approach 1:
The container image is pre-loaded with data from the storage system before actual workload execution. This preliminary action allows data to be cached locally in the container, so when workload requests arrive, the data is already available without requiring real-time network transfer, thus maintaining data availability while reducing network traffic during peak access periods
Solution Approach 2:
The container image acts as an intermediary between the storage system and the workload. Instead of workloads directly accessing data over the network from storage, they interact with the container image which has pre-loaded data. This intermediary layer handles data requests locally, reducing network traffic while ensuring data availability through the cached copies in the container
2Adaptability or versatility
If multiple users access data simultaneously, then service coverage is improved, but network bottlenecking occurs
Solution Approach 1:
The system segments data access by creating multiple container images, each loaded with specific data subsets or filtered data sets relevant to different workload types or users. This segmentation allows multiple users to access different container images simultaneously without competing for the same network bandwidth, as each container serves its data locally without requiring continuous network access to the central storage system
Solution Approach 2:
Data is pre-loaded into container images before deployment to multiple users. This preliminary action ensures that when multiple users access data simultaneously, they are retrieving data from local container caches rather than competing for network bandwidth to access the same central storage system, thereby maintaining service coverage while preventing network bottlenecking
3Ease of operation
If data is transferred to users for execution, then data access flexibility is improved, but network load and storage system load increase
Solution Approach 1:
Instead of transferring large amounts of data over the network to users for execution, the system creates container images that contain pre-loaded data and executes workloads within these containers. This copying approach allows data to be accessed flexibly within the container environment without requiring physical data transfer over the network, reducing network load and storage system load while maintaining data access flexibility
Solution Approach 2:
The container image serves as an intermediary that provides data access flexibility without requiring full data transfer. Users can execute workloads with access to the data loaded in the container, and the container manages data retrieval from storage as needed, reducing the burden on both network and storage systems compared to transferring all data upfront
4Ease of operation
If workload requests are processed through network to storage system, then data access is simplified, but network traffic increases
Solution Approach 1:
The system performs preliminary actions by loading required data into container images before workload execution. This pre-loading simplifies data access during workload execution because the data is already locally available in the container, eliminating the need for complex network requests to storage systems during runtime, thus reducing network traffic while maintaining operational simplicity
Solution Approach 2:
Container images are designed to be self-sufficient with pre-loaded data, allowing them to service workload requests without requiring continuous network communication with the storage system. This self-service capability simplifies data access within the container while dramatically reducing network traffic, as the container handles its own data needs from its internal cache
Data Source
AI summary
Management of access to data stored in a storage system is provided using container images. Specifications for a requested data scan of the storage system are received. The requested data scan is performed using a container image according to the received specifications, the container image being run on the storage system. A cost of resources is associated with the requested data scan performed using the container image. Access to data stored in the storage system is managed based on the cost of resources.