Integration User for Analytic Data Store Security and Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional analytical tools face inefficiencies when processing large transactional data sets due to complex relationships between data fields, leading to high latency and strain on infrastructure, which is unacceptable for real-time analytics applications.
Innovation Solution
The integration of a low-latency messaging protocol between transactional and analytic data store components, combined with a predicate-based row-level security scheme and separate accounting for analytic resource usage, allows for efficient data exploration and visualization by migrating query processing from servers to clients and using a directory service for worker process redundancy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional analytical tools process large transactional data sets, then data analysis capability is provided, but latency increases and infrastructure strain occurs
Solution Approach 1:
The patent segments the monolithic analytical processing system into distributed worker processes that can independently handle different query tasks. This segmentation allows parallel processing of data analysis requests, reducing overall latency while maintaining infrastructure efficiency.
Solution Approach 2:
The system pre-loads and caches frequently accessed transactional data into analytical data stores before queries are submitted. This preliminary action reduces the processing time required during actual analysis operations, thereby decreasing latency without compromising analysis capability.
2Productivity
If heavy back-end processing is performed to process standard data structures, then data transformation and modeling are achieved, but server and network infrastructure are burdened
Solution Approach 1:
The patent extracts heavy processing operations from the central server and relocates them to distributed worker processes and edge devices. This extraction reduces the computational burden on the server and network infrastructure while maintaining data transformation and modeling capabilities through distributed computing.
Solution Approach 2:
The system introduces analytical data stores as intermediary structures between transactional data sources and analysis queries. These intermediaries pre-process and transform data in advance, reducing the computational load on the server during actual query execution while maintaining transformation capability.
3Reliability
If processing is interrupted or stopped, then system reliability is maintained, but latency is incurred while waiting for processing to re-initiate
Solution Approach 1:
The patent implements continuous processing through persistent worker processes that maintain their execution state across interruptions. When processing is interrupted, these workers can resume their tasks without complete re-initialization, reducing latency while maintaining system reliability through graceful error handling and state persistence.
Solution Approach 2:
The system implements feedback mechanisms where worker processes report their status and progress to a central coordinator. This feedback enables the system to detect interruptions, manage resource allocation dynamically, and re-initiate processing efficiently while maintaining overall system reliability and minimizing latency.
4Ease of operation
If query processing is performed on servers, then centralized control is maintained, but processing speed and scalability are limited
Solution Approach 1:
The patent segments query processing functionality into independent worker processes distributed across multiple computing nodes. This segmentation enables parallel query execution and improves processing speed while maintaining centralized control through a coordinator that manages task distribution and collects results.
Solution Approach 2:
The system transitions from single-dimensional server-based processing to multi-dimensional distributed processing across multiple nodes and layers. This dimensional expansion enables parallel query execution and improves processing speed while maintaining centralized coordination through a hierarchical architecture.
Data Source
AI summary
The technology disclosed preserves the tenant specificity and user specificity of the tenant data by associating user IDs to complementary special IDs referred to as the integration user(s). In particular, it combines the traceability of user actions, the integration of security models and the flexibility of a service ID into one integration user(s).