Microservice Data Masking for Deduplicated Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data masking approaches are not well-suited for implementation in microservice deduplication environments, introducing complexities that hinder efficient data protection and confidentiality in scalable architectures.
Innovation Solution
The implementation of a microservice design with a masker controller and masker worker architecture that performs data masking inline and offline using a job approach, allowing for scalable masking operations and avoiding recalculation of fingerprints, enabling efficient data masking in deduplicated object storage environments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional data masking approaches are used, then data protection is achieved, but system complexity increases and scalability is hindered in microservice environments
Solution Approach 1:
The data masking system is divided into separate microservices: a masker controller microservice that manages masking jobs and a masker worker microservice that performs actual masking operations. This segmentation allows each component to be independently deployed, scaled, and maintained, reducing overall system complexity while maintaining reliable data protection through specialized functionality.
Solution Approach 2:
A masker controller microservice acts as an intermediary between the data ingestion pipeline and the masking operations. It receives data, manages the masking workflow, coordinates with storage systems, and handles error processing. This intermediary layer simplifies the architecture by centralizing masking logic and isolating complexity from the core data protection mechanisms.
2Reliability
If data masking is performed inline, then data confidentiality is improved, but computational overhead increases
Solution Approach 1:
The system dynamically adjusts masking behavior based on data characteristics and job requirements. The masker controller can configure masking algorithms, select appropriate workers, and adjust processing parameters in real-time. This dynamic approach optimizes computational overhead by applying masking only where necessary and using efficient algorithms for specific data types, while maintaining strong data confidentiality through flexible masking strategies.
Solution Approach 2:
The system allows configuration of masking parameters such as algorithm selection, processing granularity, and performance priorities. By changing these parameters based on the specific masking job requirements, the system achieves data confidentiality with optimized computational overhead, balancing security needs against processing resource consumption.
3Reliability
If conventional masking methods are used, then data protection is provided, but adaptability to microservice architectures is poor
Solution Approach 1:
The masker controller and masker worker microservices are designed with universal interfaces that can integrate with various storage systems, data formats, and microservice frameworks. The system supports multiple masking algorithms, data types, and deployment configurations, making it adaptable to different microservice architecture patterns while maintaining reliable data protection through consistent masking functionality.
Solution Approach 2:
The system performs preliminary configuration and setup through the masker controller, which establishes masking policies, identifies data requiring protection, and prepares masking jobs before actual processing begins. This preliminary action enables the system to adapt to microservice architectures by pre-configuring integration points and workflows, allowing seamless incorporation into existing microservice environments without requiring changes to core data protection mechanisms.
4Use of energy by moving object
If offline masking jobs are used, then computational resources are optimized, but processing time increases
Solution Approach 1:
The system implements periodic masking job execution where the masker controller schedules and monitors masking operations at intervals. Offline masking jobs can be queued and executed periodically rather than immediately, optimizing computational resource utilization by batch-processing similar operations. This periodic action reduces overall processing time by consolidating tasks and utilizing idle computational resources efficiently.
Solution Approach 2:
The masker controller maintains continuous monitoring and management of masking jobs, ensuring that computational resources are continuously allocated to active masking tasks. By keeping the masking pipeline continuous and managing job queues efficiently, the system minimizes idle time and ensures that computational resources are always being productively used, balancing resource optimization with acceptable processing times through continuous workflow management.
Data Source
AI summary
A method includes receiving an object from a client as part of a data ingestion process, directing the object to an object access microservice, providing a copy of the object to a masker worker microservice, masking the copy of the object to create a masked object, and the masking of the copy of the object is performed inline prior to storage of the object, deduplicating the object, and storing the masked object and the object in storage.