Inverted Text Index for Sensitive Data Masking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data masking methods are resource-intensive and inefficient when dealing with diverse and dynamic data sets, particularly in cloud environments, as they often require individual processing of each document and extensive infrastructure setup.
Innovation Solution
The method employs an inverted text index to evaluate statistical measures of index tokens, select candidate tokens potentially containing sensitive information, and mask specific document tokens based on associated metadata, using a classifier to ensure compliance with governance rules, thereby optimizing processing and reducing resource consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional data masking methods process each document individually, then data protection accuracy is maintained, but processing time and resource consumption increase significantly
Solution Approach 1:
The patent merges multiple document processing operations into a single batch processing workflow. By collecting multiple documents and processing them together through a unified masking pipeline, the system reduces redundant operations and achieves significant time savings while maintaining protection accuracy through consistent application of masking rules across all documents.
Solution Approach 2:
The system performs preliminary actions by pre-identifying and cataloging sensitive information patterns, masking rules, and replacement strategies before actual document processing begins. This preparation phase includes building lookup tables of sensitive data types and configuring masking parameters in advance, which accelerates the subsequent batch processing while ensuring accurate protection.
2Measurement precision
If comprehensive statistical analysis is performed on all tokens, then sensitive information identification accuracy improves, but computational resources and processing time increase
Solution Approach 1:
The patent applies local quality by performing statistical analysis selectively rather than uniformly across all tokens. The system identifies and applies intensive statistical measures only to tokens that exhibit characteristics of potential sensitive information, while using lighter-weight methods for common non-sensitive tokens. This differentiated approach maintains high identification accuracy for sensitive data while reducing overall computational resource consumption.
3Manufacturing precision
If detailed metadata extraction and association is performed for all candidate tokens, then masking precision improves, but processing complexity and time increase
Solution Approach 1:
The system applies partial action by extracting and associating metadata only for candidate tokens that meet specific criteria for being sensitive information. Rather than performing complete metadata extraction on all tokens, the system selectively processes only those tokens that have a reasonable probability of being sensitive based on initial filtering, thereby achieving sufficient masking precision while avoiding unnecessary processing complexity.
Data Source
AI summary
The exemplary embodiments disclose a method, a computer program product, and a computer system for protecting sensitive information. The exemplary embodiments may include using an inverted text index for evaluating one or more statistical measures of an index token of the inverted text index, using the one or more statistical measures for selecting a set of candidate tokens, extracting metadata from the inverted text index, associating the set of candidate tokens with respective token metadata, tokenizing at least one document resulting in one or more document tokens, comparing the one or more document tokens with the set of candidate tokens, selecting a set of document tokens to be masked, selecting at least part of the set of document tokens that comprises sensitive information according to the associated token metadata, masking the at least part of the set of document tokens, and providing one or more masked documents.