Distributed State Store for Multi-Protocol Container Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Container state is lost when pods are moved or terminated, and existing solutions like CSI do not adequately support access through multiple protocols, especially in ML pipelines with diverse protocol requirements.
Innovation Solution
A distributed state store allows access to container state via multiple protocols, using gateway pods to translate between unsupported and supported protocols, enabling storage and retrieval across a Kubernetes cluster.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If container state is externalized via CSI, then state persistence is improved, but protocol flexibility deteriorates
Solution Approach 1:
The data fabric pod is designed to provide multiple protocol interfaces (HDFS, POSIX, S3, HBase, JDBC, OJAI, streaming) within a single unified storage system. This allows the same underlying storage infrastructure to serve diverse protocol requirements across different ML pipeline tools, eliminating the need for separate storage solutions for each protocol type while maintaining state persistence reliability.
Solution Approach 2:
The data fabric pod acts as an intermediary layer between containerized ML tools and the underlying distributed storage system. It translates various protocol requests into unified storage operations, enabling tools with different protocol requirements to access the same persistent state without requiring protocol-specific storage implementations.
2Adaptability or versatility
If multiple protocol interfaces are added to data fabric pods, then protocol compatibility is improved, but pod complexity increases
Solution Approach 1:
Multiple protocol interfaces (HDFS, POSIX, S3, HBase, JDBC, OJAI, streaming) are merged into a single unified data fabric pod implementation. Rather than deploying separate storage services for each protocol, the system combines all protocol support within one pod type, reducing overall system complexity while maintaining broad protocol compatibility.
Solution Approach 2:
The data fabric pod is designed as a universal storage service that can handle multiple protocols simultaneously. This multi-functional design eliminates the need for multiple specialized pods, reducing deployment complexity and making the system easier to manage while supporting diverse protocol requirements.
3Ease of operation
If protocol translation is implemented, then access to container state is improved, but system complexity increases
Solution Approach 1:
The data fabric pod serves as an intermediary that automatically translates between various protocols and the underlying storage system. This translation layer is built into the pod's core functionality, making protocol translation transparent to users and eliminating the need for complex external translation services while improving ease of access to container state.
Solution Approach 2:
The data fabric pod performs protocol translation autonomously without requiring external translation services or manual configuration. The pod itself provides the translation capability as an integrated feature, making the system self-sufficient and reducing overall complexity by eliminating separate translation components.
Data Source
AI summary
Examples described herein are generally directed to facilitating access to container state through multiple protocols. According to an example, a distributed state store is provided in which container state is accessible by multiple client containers via nodes of the distributed state store across which the container state is replicated. Responsive to receipt from a first client container of a storage request via a first application programming interface (API) method of the distributed state store that is associated with a first protocol, the container state is stored within the distributed state store. Responsive to receipt from a second client container of an access request via a second API method of the distributed state store that is associated with a second protocol, the container state is returned to the second client container via the second protocol.