Event Message Queue Redaction for Sensitive Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for redacting sensitive data in large-scale storage environments are inefficient, requiring substantial computational resources and leading to delays in identifying and protecting personally identifiable and financial information, which can result in exposure to unauthorized access as the scale of data increases.
Innovation Solution
A system that generates real-time event messages for changed data objects, which are processed in message queues to determine and immediately redact sensitive data types based on predefined rules, using deep data inspection and classification techniques to identify facets indicating data to be redacted, ensuring immediate compliance with regulations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If periodic scanning of source storage systems is used to determine data for redaction, then comprehensive data inspection is achieved, but processing delays and computational resource consumption increase significantly
Solution Approach 1:
The system performs preliminary action by generating event messages that contain pre-computed facets about data instances before redaction processing is needed. These event messages are published to message queues in advance, allowing the redaction system to immediately process and act on them without waiting for periodic scans or deep inspection at redaction time.
Solution Approach 2:
The system extracts only the necessary information (facets about data instances) from the original data objects and places them into event messages in the message queue. This extraction eliminates the need to repeatedly scan and re-inspect the entire source storage systems, reducing computational overhead while maintaining inspection completeness.
2Measurement precision
If deep data inspection is performed on source storage systems to identify sensitive data, then accurate identification of redaction candidates is achieved, but computational resources and processing time increase exponentially with data scale
Solution Approach 1:
The system extracts only the essential facets (sensitive data characteristics) from complete data objects and stores them in event messages. This allows the redaction system to work with lightweight message structures rather than inspecting entire large-scale data objects, maintaining identification accuracy while dramatically improving processing throughput.
Solution Approach 2:
Instead of working with the original large data objects, the system creates simplified copies in the form of event messages containing only the relevant facets. These message copies can be rapidly processed, inspected, and matched against redaction rules without the computational burden of handling the full data objects.
3Reliability
If periodic scanning is used to update redaction data, then compliance with redaction policies is maintained, but real-time protection against unauthorized access is compromised
Solution Approach 1:
The system implements continuous action by maintaining an always-active message queue that receives and processes event messages in real-time. As soon as new data is added or modified in the source storage system, an event message is immediately generated and processed, ensuring continuous compliance monitoring and real-time redaction response without periodic interruptions.
Solution Approach 2:
The system establishes a feedback loop where changes to data objects trigger event messages that are immediately processed to determine if redaction is needed. This closed-loop feedback mechanism ensures that redaction policy compliance is automatically maintained in real-time, with the system responding immediately to any changes that may require redaction.
Data Source
AI summary
Provided are a computer program product, method, and system for determining data to redact in an object. Event messages for objects are received. Each event message includes at least one facet, each facet for an instance of data in an object resulting from deep data inspection of the object that is used to determine whether the instance of data should be redacted. The event messages are added to a message queue. A determination is made of an event message in the message queue for an object having a facet for an instance of data that indicates the instance of data is to be redacted. Redaction is performed on the instance of data in the object in the determined event message having the facet indicating the instance of data is to be redacted data.