Virtual Identifier Policy Enforcement for Distributed Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The complexity of policy enforcement in distributed storage and processing environments, such as those using Apache Hadoop, is increased due to the variety of access methods, requiring separate policy rules for each data source and access method, making it difficult to manage and enforce access control consistently.
Innovation Solution
Assigning virtual identifiers to blocks of files or columns of tables across different data sources allows for the application of access control policies, which are then enforced across the data sources, simplifying policy enforcement by using a common concept expressed differently in various sources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If separate policy rules are created for each data source and access method, then policy enforcement can be applied to each specific source, but the complexity of policy management increases significantly
Solution Approach 1:
The patent introduces virtual identifiers that serve as universal keys across multiple data sources and access methods. Instead of creating separate policy rules for each data source (HDFS, relational databases, NoSQL databases) and access method (Hive, Spark, Solr), the system maps all these diverse sources to common virtual identifiers. This allows a single policy rule set to enforce access control consistently across the entire distributed ecosystem, eliminating the need for redundant policy definitions while maintaining comprehensive coverage.
Solution Approach 2:
The virtual identifier acts as an intermediary layer between the diverse data sources/access methods and the policy enforcement mechanism. The mapping module translates real names from various data sources into virtual identifiers, which then serve as the common interface for policy application. This intermediary abstraction decouples the complexity of multiple data sources from the policy enforcement logic, simplifying management while ensuring consistent policy application across all sources.
2Adaptability or versatility
If multiple access methods are supported for distributed storage and processing, then data accessibility and processing flexibility improve, but the difficulty of enforcing access control policies increases
Solution Approach 1:
The virtual identifier system provides a universal interface that works across all access methods (Hive, Spark, Solr) and data sources. Each access method continues to operate independently with its own query language and interface, but they all map to the same virtual identifiers. This allows the system to support multiple access methods and data sources with full versatility while maintaining simplified policy enforcement through the common virtual identifier namespace.
Data Source
AI summary
A non-transitory computer readable storage medium has instructions executed by a processor to assign virtual identifiers to blocks of a file that contain identical information in different data sources. A distributed storage and distributed processing query statement is received. Real name attributes of the query statement are equated with selected virtual identifiers. Access control policies are applied to the selected virtual identifiers to obtain policy results. The policy results are applied to the real name attributes of the query statement to obtain query results.