In-Memory Database Intelligent Unmasking for Batch Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In-memory database architectures face performance delays and reduced efficiency when handling masked data due to the need for application logic to be pushed down to the database layer, which complicates batch processing.
Innovation Solution
The implementation of an unmasking process that identifies mask intervals, performs joins with unmasked elements sharing a common key, groups the results, and further joins to produce an unmasked table, allowing efficient batch processing by leveraging the in-memory database engine's processing power.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If application logic is pushed down to the database layer to handle masked data, then data retrieval capability is improved, but processing speed deteriorates
Solution Approach 1:
The patent performs unmasking operations in advance before the main data processing occurs. The engine identifies mask intervals and performs joins with unmasked elements beforehand, converting masked data into unmasked data prior to batch processing. This preliminary action eliminates the need for complex application logic during execution, maintaining both versatility and speed.
Solution Approach 2:
The patent introduces an intermediary unmasking process that acts as a bridge between the application layer and the database engine. This intermediary layer handles the complexity of masked data interpretation, allowing the database engine to focus on efficient batch processing without being burdened by application-specific masking logic.
2Productivity
If batch processing is used to increase speed, then processing efficiency is improved, but handling of masked data becomes more complex
Solution Approach 1:
The unmasking process is performed in advance of batch processing, converting all masked data into unmasked form before the efficiency-critical batch operations begin. This eliminates the need to handle masking complexity during batch processing, allowing the system to leverage full batch processing efficiency.
Solution Approach 2:
The patent segments the data processing into distinct phases: first identifying mask intervals and performing preliminary joins, then grouping by mask position, and finally executing efficient batch processing on the unmasked data. This segmentation isolates the complexity to a preliminary phase, allowing the main batch processing phase to operate simply and efficiently.
3Ease of operation
If conventional disk-based database architecture is used, then data retrieval is simplified, but processing speed is reduced
Solution Approach 1:
The patent replaces the conventional disk-based database mechanical access model with an in-memory database engine that leverages memory-based processing. By performing unmasking operations within the memory-based engine using vectorized operations and batch processing, the system achieves both the simplicity of conventional approaches and the speed of memory-based processing.
Solution Approach 2:
The patent changes the fundamental parameter of data storage location from disk to memory, and changes the processing model from row-by-row to batch processing. These parameter changes enable the system to maintain operational simplicity while achieving dramatic speed improvements through in-memory batch operations.
Data Source
AI summary
Database data is unmasked in order to facilitate its efficient handling by a database engine. In response to a request for data of a masked table including a masked element, an engine identifies a mask interval, and then performs a first join with unmasked elements sharing a common key. The table resulting from this first join is then grouped according to a highest level location of the mask. A second join is then performed between the results of this grouping and the mask interval, to produce a corresponding unmasked table including a plurality of unmasked elements corresponding to the masked element. Unmasking according to embodiments may be particularly useful in leveraging processing power of an in-memory database engine, allowing it to efficiently perform batch processing of requests for masked data received from software of an overlying application layer.