Data Lake Access Control via Metadata Linkage and Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data lakes face challenges in scaling access control mechanisms, particularly due to the vast number of data sets and diverse data sources, which complicates fine-grained access management for different data consumers.
Innovation Solution
The proposed solution involves enriching data upon replication to a data lake by incorporating data domain information and access control metadata. This enriched data is then used to model data linkages, allowing for the derivation of new access models. Access control is granted based on metadata, linkage relationships, and context information, enabling selective and granular access for data consumers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional access control lists (ACLs) are used to manage data access in data lakes, then access control can be implemented, but the system becomes difficult to scale and maintain due to the vast number of data sets and diverse data sources
Solution Approach 1:
The patent segments access control into multiple layers: data domain level (coarse-grained) and data object level (fine-grained). This segmentation allows the system to manage access control more efficiently by dividing the complex task into manageable parts, reducing overall system complexity while maintaining reliability.
Solution Approach 2:
The patent introduces data domain information and metadata as intermediary elements between data consumers and data objects. These intermediaries enable automated access control decisions by providing context about data relationships, origins, and sensitivity, reducing the need for manual ACL management.
2Reliability
If fine-grained access control is implemented for each data object in the data lake, then data security is improved, but the effort to design and maintain access control increases significantly
Solution Approach 1:
The patent performs preliminary actions by enriching data with metadata and domain information during the data ingestion phase, before access control decisions are needed. This preliminary enrichment automates much of the access control configuration, reducing the effort required for design and maintenance while maintaining fine-grained security.
Solution Approach 2:
The system enables self-service access control by allowing data domains and metadata to automatically determine access permissions based on predefined policies and data relationships. This reduces manual intervention and maintenance effort while ensuring consistent security enforcement across all data objects.
3Adaptability or versatility
If data consumers are granted broad access to data in the data lake, then data utility and analysis capability are improved, but data security and access control become more difficult to manage
Solution Approach 1:
The patent implements dynamic access control where permissions are not static but adapt based on data domain information, metadata, and context. This allows data consumers to receive appropriate access levels dynamically determined by the system, providing flexibility while managing complexity through automated decision-making.
Solution Approach 2:
The system changes access control parameters based on data domain characteristics, metadata attributes, and consumer context. By varying access parameters dynamically rather than using fixed permissions, the system achieves flexibility in data access while reducing management complexity through rule-based automation.
Data Source
AI summary
In an example embodiment, access to a data set in a data lake can be specified using several approaches, based on the metadata and information attached. The metadata may be replicated from the original data source of the underlying data, and additional metadata may be modeled and stored to construct linkage information between data types. This linkage information may be used to automatically grant access to users to additional objects that are linked to objects that the user has explicit access to.