Cloud Data Pipeline Access Control with Unit-Level Runtime Accounts
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data processing systems face challenges in managing complex data pipelines on cloud platforms due to monolithic architectures, which require extensive expertise, expose large amounts of data when user accounts are compromised, and lead to significant data corruption risks.
Innovation Solution
A system that configures and executes data pipelines on cloud platforms using declarative specifications, implements a least privilege policy by creating separate runtime system accounts for each data pipeline unit, and manages access control to limit data access, enabling modular upgrades and decentralized ownership.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If monolithic architecture is used for data processing, then data processing capability is improved, but system complexity and difficulty of management increase
Solution Approach 1:
The patent divides the monolithic data processing system into multiple independent data pipeline units, each handling specific data processing tasks. These units can be independently configured, deployed, and managed, reducing overall system complexity while maintaining processing capability through modular composition.
Solution Approach 2:
The patent introduces a new architectural dimension by implementing data pipelines as multi-stage processing flows where data passes through multiple independent units sequentially. This transforms the traditional monolithic structure into a distributed pipeline architecture, enabling independent management of each unit while maintaining overall processing capability.
2Ease of operation
If users have broad access to data for development, then ease of operation is improved, but security and data exposure risks increase
Solution Approach 1:
The patent implements differentiated access control where each data pipeline unit has its own dedicated service account with permissions scoped to that specific unit. This ensures that even if one account is compromised, the damage is limited to a single pipeline unit rather than the entire data system, while developers can still access and modify their assigned units freely.
Solution Approach 2:
The patent introduces service accounts as intermediary entities between users and data pipeline units. These service accounts act as mediators that provide controlled access to pipeline units, enabling developers to work with data while preventing direct access to the underlying data infrastructure, thus reducing exposure risk.
3Ease of operation
If single service account is used for data pipeline execution, then ease of operation is improved, but reliability and security decrease
Solution Approach 1:
The patent segments the single service account into multiple dedicated service accounts, each assigned to a specific data pipeline unit. This segmentation enables isolated failure domains where compromises or errors in one unit do not affect others, improving reliability while maintaining operational simplicity through automated account management.
Solution Approach 2:
The patent dynamically changes the permission parameters of service accounts based on the specific data pipeline unit they serve. Each service account is configured with permission sets tailored to its associated pipeline unit's requirements, providing appropriate access levels while minimizing exposure, thus improving security without complicating operations.
Data Source
AI summary
A system performs continuous delivery of a data pipeline on a cloud platform. The system receives a specification of the data pipeline comprising data pipeline units. The system generates a deployment package for each data pipeline unit for a cloud platform. The system provisions computing infrastructure on the cloud platform according to the system configuration of the data pipeline unit. The data pipeline may be implemented as a data mesh. The data pipeline generates one or more data models. The system receives a schema representing a modification in a data model based on a change in the requirements of a consumer system. The system determines the changes to the data pipeline based on the received schema and reconfigures the data pipeline to generate the modified data model. The system manages access control of data to minimize the exposure to data in case of accidental or malicious data breach.