Multitenant ETL Pipeline Orchestration with DAG-Specific IAM
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing ETL environments face challenges in managing multiple directed acyclic graphs (DAGs) due to limited default identity and access management (IAM) roles, leading to unintended resource access and increased complexity and cost when multiple instances are spun up.
Innovation Solution
Implementing individual IAM role permissions for each DAG within an ETL instance, using a plugin to manage and validate user code permissions, allowing secure execution of multiple DAGs in a single multitenant environment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If a default IAM permissions role is used to allow access to all DAGs in an ETL instance, then ease of operation is improved, but security deteriorates as one DAG can access the resources of another DAG
Solution Approach 1:
The patent segments the monolithic default IAM role into individual DAG-specific IAM roles. Each DAG is assigned its own IAM role with permissions scoped to only the resources it needs, preventing one DAG from accessing another DAG's resources while maintaining ease of operation through automated role assignment.
Solution Approach 2:
The patent implements local quality by making IAM permissions specific to each DAG rather than applying a uniform default role. Each DAG receives tailored permissions that are locally optimized for its specific resource access needs, enhancing security without compromising operational ease through the system's automated permission management.
2Reliability
If multiple instances of the ETL process are spun up to isolate DAGs, then security is improved, but device complexity increases as DAGs may require cross-instance access
Solution Approach 1:
The patent merges multiple isolated ETL instances into a single multitenant instance by implementing fine-grained IAM roles for each DAG. This allows multiple DAGs to coexist securely in one instance with proper permission boundaries, eliminating the need for multiple separate instances and their associated cross-instance access complexities.
Solution Approach 2:
The patent creates a composite permission structure that combines multiple DAG-specific IAM roles within a single ETL instance. This composite approach allows the system to achieve the security isolation of multiple instances while maintaining the operational simplicity of a single instance, as each DAG's permissions are compositely managed through its own IAM role.
3Adaptability or versatility
If additional ETL process instances are provisioned to manage multiple DAGs, then adaptability is improved, but loss of energy increases due to additional CPU, memory, and network resources
Solution Approach 1:
The patent makes a single ETL instance universal by enabling it to securely host and manage multiple DAGs through individual IAM roles. This multi-functional approach allows one instance to adapt to various DAG workloads and permission requirements without provisioning additional instances, thereby reducing energy consumption while maintaining adaptability.
Solution Approach 2:
The patent changes the permission parameter from a monolithic default role to granular DAG-specific roles, allowing a single ETL instance to efficiently manage multiple DAGs with different resource access requirements. This parameter change enables the system to adapt to diverse DAG needs without the energy cost of multiple separate instances.
Data Source
AI summary
Certain aspects of the present disclosure provide techniques for multitenancy pipeline orchestration. According to certain embodiments, a user stores user code in a code repository and defines user code permissions needed to execute the code in an ETL system, that are stored in a database on a processing platform. The user code is imported to the ETL environment, and a plugin obtains the user code permissions from the processing platforms and generates a DAG having an operational context that includes the user code and user code permissions, that govern access permissions during execution on an ETL engine. When execution of the DAG ceases, the user code and user code permissions are removed from the ETL system, according to certain embodiments.