Containerized Data Ingestion and Processing Framework for Cloud Scalability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional enterprise-level data moving technologies are not scalable and do not fit well in dynamic provisioning, highly available cloud environments, as they are designed in a tightly-coupled manner and lack support for stateful data processing tasks and cloud-native applications.
Innovation Solution
A micro-service oriented data processing and data moving framework is implemented, utilizing containerized components and a scalable architecture that includes a staging cluster and a processing cluster, with agents, brokers, and security components, enabling flexible data ingest, deduplication, compression, and encryption, and supporting multiple data sources and applications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If traditional enterprise-level data moving technologies are used, then data compression and deduplication features are provided, but scalability and adaptability to cloud environments deteriorate
Solution Approach 1:
The system segments data movement functionality into independent containerized components (data ingest components, broker components, data processing components) that can be independently deployed, scaled, and managed in cloud environments. Each component handles specific tasks such as data ingestion, brokering, processing, and transmission, allowing the system to maintain enterprise-level features while adapting to cloud scalability requirements.
Solution Approach 2:
The containerized architecture creates universal components that can function across different cloud environments and data movement scenarios. The broker components serve multiple functions including queuing, routing, and coordination, while the standardized container interface allows the same components to operate in various cloud platforms, achieving both feature richness and environmental adaptability.
2Productivity
If traditional tightly-coupled data moving architectures are used, then efficient data processing is achieved, but scalability and fault-tolerance deteriorate
Solution Approach 1:
The tightly-coupled architecture is segmented into loosely-coupled containerized components that communicate through standardized interfaces. Data ingest components feed data to broker components, which queue and route to processing components, which then transmit to target sites. This segmentation maintains processing efficiency while enabling independent failure isolation and component replacement, thus improving fault-tolerance and scalability.
Solution Approach 2:
Broker components serve as intermediaries between data ingest components and data processing components, providing queuing and routing functionality. This intermediary layer decouples the direct connection between ingestion and processing, allowing independent scaling and failure handling of each component type while maintaining overall system efficiency through coordinated operation.
3Adaptability or versatility
If containerized components with scalable architecture are implemented, then adaptability to cloud environments is improved, but system complexity increases
Solution Approach 1:
The system divides complex data movement functionality into standardized containerized components with well-defined interfaces. Each container encapsulates specific functionality (ingestion, brokering, processing, transmission) and can be independently deployed and managed. This segmentation reduces operational complexity by allowing independent lifecycle management of each component while maintaining cloud environment adaptability through container orchestration.
Solution Approach 2:
The containerized architecture enables copying and replication of identical component instances across multiple nodes and locations. Standardized container images can be replicated to create multiple ingest components, broker components, or processing components, simplifying deployment and scaling operations while maintaining system adaptability to various cloud environments through consistent container interfaces.
Data Source
AI summary
A plurality of data sets to be moved from a source site to a target site in a cloud computing platform is received at a plurality of a containerized data ingest components located at the source site. The received plurality of data sets are provided from the plurality of data ingest components to a staging cluster comprising a plurality of containerized broker components located at the source site, wherein the plurality of containerized broker components queue the plurality of data sets. The queued plurality of data sets are provided from the plurality of containerized broker components to a processing cluster comprising a plurality of containerized data processing components, wherein the plurality of containerized data processing components process the plurality of data sets. The plurality of data sets is transmitted from the plurality of containerized data processing components to the target site.