Predicate-Based Row Level Security for Analytical Data Stores
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional analytical tools face inefficiencies when processing large transactional data sets due to complex relationships between data fields, leading to high latency and strain on infrastructure, which is unacceptable for real-time analytics applications.
Innovation Solution
The integration of a low-latency messaging protocol between transactional and analytic data store components, combined with a predicate-based row-level security scheme and separate resource usage accounting, allows for efficient data processing and secure data access, enabling fast data exploration and visualization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional analytical tools are used to process large transactional data sets, then data analysis can be performed, but processing latency is high and infrastructure strain increases
Solution Approach 1:
The patent segments the monolithic analytical processing system into distributed microservices that can independently process different portions of data in parallel. This segmentation enables concurrent processing of large transactional data sets, reducing overall processing latency while maintaining high productivity across the distributed infrastructure.
Solution Approach 2:
The patent introduces a temporal dimension to data processing by implementing real-time streaming capabilities alongside batch processing. This allows the system to handle data analysis across multiple time dimensions simultaneously, reducing latency for time-sensitive queries while maintaining comprehensive analytical productivity.
2Productivity
If extensive compute resources are allocated for processing, then data analysis capability is maintained, but infrastructure burden and cost increase
Solution Approach 1:
The patent implements universal data structures and processing pipelines that can handle multiple analytical workloads using the same infrastructure. This multi-functionality allows a single compute resource to serve various analytical purposes simultaneously, reducing the total quantity of compute resources needed while maintaining high analytical productivity across different data types and query patterns.
Solution Approach 2:
The patent dynamically adjusts processing parameters such as batch sizes, parallelism levels, and memory allocation based on workload characteristics. This adaptive parameter tuning optimizes resource utilization for each specific analytical task, maximizing productivity while minimizing the quantity of compute resources consumed for each operation.
3Loss of time
If real-time processing is implemented, then latency is reduced, but system complexity and resource requirements increase
Solution Approach 1:
The patent implements preliminary data preprocessing and transformation pipelines that prepare data in advance for analytical queries. By performing data cleaning, normalization, and aggregation before analysis requests arrive, the system reduces real-time processing complexity while maintaining low latency data availability through pre-computed results and optimized data structures.
Data Source
AI summary
A predicate-based row level security system is used when workers build or split an analytical data store. According to one implementation, predicate-based means that security requirements of source transactional systems can be used as predicates to a rule base that generates one or more security tokens, which are associated with each row as attributes of a dimension. Similarly, when an analytic data store is to be split, build job, user and session attributes can be used to generate complementary security tokens that are compared to security tokens of selected rows. Efficient indexing of a security tokens dimension makes it efficient to qualify row retrieval based on security criteria.