Low Latency Directory Service for Transactional Data Integration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional analytical tools face inefficiencies when processing large transaction data sets due to complex relationships between data fields, leading to time-consuming backend processing that incurs latency, particularly in real-time analytics applications, straining server and network infrastructure.
Innovation Solution
The integration of a low-latency messaging protocol between transactional and analytic data store components, along with a predicate-based row-level security scheme and separate resource usage accounting, enables efficient data processing and visualization by migrating query processing from servers to clients and using purpose-designed queues and status reporting channels.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional analytical tools process large transaction data sets, then data analysis capability is provided, but processing time and latency increase significantly
Solution Approach 1:
The patent segments the monolithic backend processing system into multiple worker threads that can process different portions of data sets concurrently. This parallelization reduces overall processing time while maintaining full data analysis capability, directly resolving the contradiction between productivity and time loss.
2Reliability
If standard data structures are used for transaction data, then data integrity is maintained, but compute resources required for processing increase
Solution Approach 1:
The patent transforms the data structure parameters by converting traditional relational data models into columnar storage formats with optimized schemas for analytical queries. This parameter change reduces the compute resources required for processing while maintaining data integrity through structured storage and indexing mechanisms.
3Productivity
If heavy backend processing is performed, then data transformation and analysis are achieved, but server and network infrastructure are strained
Solution Approach 1:
The patent extracts heavy processing operations from the central server to distributed worker threads that operate independently. This extraction reduces the computational burden on the main server and network infrastructure while maintaining full data transformation capability through the distributed worker pool.
4Stability of the object's composition
If processing is interrupted or stopped, then system stability is maintained, but latency increases while waiting for processing to re-initiate
Solution Approach 1:
The patent implements continuous processing through a pool of worker threads that can pick up and continue processing tasks after interruptions. The system maintains stability through structured error handling and task queuing while minimizing latency by having pre-positioned worker threads ready to resume operations immediately without full system re-initialization.
Data Source
AI summary
Low latency communication between a transactional system and analytic data store resources can be accomplished through a low latency key-value store with purpose-designed queues and status reporting channels. Posting by the transactional system to input queues and complementary posting by analytic system workers to output queues is described. On-demand production and splitting of analytic data stores requires significant elapsed processing time, so a separate process status reporting channel is described to which workers can periodically post their progress, thereby avoiding progress inquiries and interruptions of processing to generate report status. This arrangement produces low latency and reduced overhead for interactions between the transactional system and the analytic data store system.