Query Language Conversion for Big Data Access Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Big data environments face challenges in access control, particularly with changing table structures, where traditional relational databases struggle to handle large volumes of data and managing permissions across various data storage formats, leading to duplicative data and difficulties in enforcing column and row-level access control.
Innovation Solution
A system that detects the language type of a query, generates a temporary table, and applies access permissions by validating queries against a permissions database, ensuring atomicity, consistency, isolation, and durability, while masking sensitive information and populating unavailable columns with null values.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional relational databases are used to manage big data, then access control and permission management are supported, but the system cannot handle large volumes of data and changing table structures efficiently
Solution Approach 1:
The system segments the monolithic database structure into multiple data storage formats (relational databases, flat files, data lakes) that can be independently managed. Each format can be optimized for specific access control requirements while the overall system maintains adaptability through the unified access control layer that manages permissions across all segmented storage formats.
2Adaptability or versatility
If data is stored in multiple data storage formats for different use cases, then data accessibility and versatility are improved, but managing access permissions across various formats becomes difficult
Solution Approach 1:
The access control system is designed as a universal multi-functional platform that can manage permissions across relational databases, flat files, and data lakes through a single unified interface. The system provides table-level, column-level, and row-level access control that works consistently across all storage formats, eliminating the need for separate permission management systems for each format.
3Adaptability or versatility
If columns are added or deleted in big data storage formats, then data structure adaptability is improved, but duplicative copies must be created consuming terabytes of storage
Solution Approach 1:
The system performs preliminary actions by creating virtual views and metadata descriptors that define table schemas before actual data operations. When columns are added or deleted, the system updates the metadata and virtual views rather than physically copying entire terabyte-sized tables. This allows schema flexibility to be achieved through lightweight metadata updates instead of massive data duplication.
4Reliability
If row-level access control is implemented, then data security is improved, but query processing complexity increases
Solution Approach 1:
The system performs preliminary actions by pre-computing and caching access control policies, row-level permissions, and data classification metadata before query execution. When a query is submitted, the system has already prepared the necessary access control filters and can apply them efficiently during query processing rather than computing permissions in real-time, thus maintaining both security and performance.
Data Source
AI summary
The present disclosure involves detecting a first language type of a query configured to access a column in a data table, wherein the query is requested by a user; converting, by the computing device, the query from the first language type to a second language type that conforms with the data storage format of the data table; after converting the query to the second language type, generating, by the computing device, a temporary table including the column and enforcing an access permission by formatting the temporary table to support the second language type and excluding a row from the temporary table for which the user does not have access rights; and executing, by the computing device, the query against the temporary table to generate a query result.