Text Sanitization via K-Anonymity and Frequency Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing document sanitization techniques lack generality and scalability, struggling to automatically identify and redact sensitive information in heterogeneous documents, often over- or under-redacting data, which affects the usefulness of training data for automated systems like ASR.
Innovation Solution
A system and method that employs three steps: filtering tokens against a privacy threshold, performing k-anonymity sanitization, and using a machine-learning language model to replace or redact unsafe tokens, with a heuristic approach to efficiently find a near-optimal k-value for k-anonymity, ensuring data is sanitized while remaining useful for training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If known document sanitization techniques are used to detect specific information from concrete domains, then sensitive information can be removed, but the system lacks generality and scalability across heterogeneous documents
Solution Approach 1:
The patent applies universality by creating a domain-agnostic sanitization system that processes heterogeneous documents across multiple domains (medical, legal, financial, etc.) using a single unified approach. The system converts documents from various domains into standardized document vectors and applies the same frequency-based filtering and k-anonymity algorithms regardless of the original domain, eliminating the need for domain-specific customization while maintaining effective sensitive information removal.
Solution Approach 2:
The patent transforms the sanitization problem from domain-specific pattern matching to a parameter-based approach using document frequency thresholds and k-anonymity parameters. By converting documents into numerical vectors and applying mathematical operations (frequency counting, threshold comparison, k-anonymity enforcement), the system achieves domain independence while maintaining reliability in identifying and removing sensitive information across heterogeneous document types.
2Reliability
If a system over-redacts data to ensure privacy, then sensitive information is removed, but the remaining data becomes less useful for training purposes
Solution Approach 1:
The patent applies partial action by selectively redacting only the tokens that meet the privacy threshold criteria rather than applying blanket redaction rules. The frequency-based filtering identifies and redacts only those tokens below the threshold, leaving frequent (non-sensitive) tokens intact. This partial approach maintains privacy protection while preserving data utility, avoiding the excessive redaction that would render training data useless.
Solution Approach 2:
The system uses feedback mechanisms through the frequency analysis loop, where token frequencies are calculated, compared against thresholds, and used to determine which tokens to redact. The k-anonymity enforcement also provides feedback by iteratively checking whether redacted documents meet the anonymity requirement, adjusting redaction decisions accordingly. This feedback-driven approach optimizes the balance between privacy protection and data utility.
3Loss of information
If a system does not redact enough data, then data utility is maintained, but sensitive information may remain in the documents
Solution Approach 1:
The patent applies preliminary action by pre-calculating token frequencies across the entire corpus before performing redaction. This preliminary frequency analysis establishes a baseline that guides subsequent redaction decisions, ensuring that tokens below the privacy threshold are identified and removed before the actual sanitization process. This preliminary preparation enables the system to maintain data utility while preventing sensitive information exposure through systematic threshold-based filtering.
Solution Approach 2:
The patent introduces frequency thresholds and k-anonymity parameters as intermediary mechanisms that mediate between data utility and privacy protection. These intermediaries translate the abstract goal of privacy preservation into concrete, actionable redaction criteria, enabling the system to automatically determine which tokens to redact without compromising either data utility or sensitive information protection.
4Measurement precision
If linear programming approach is used to find optimal k vector for k-anonymity, then optimal solution can be found, but computational cost becomes very expensive
Solution Approach 1:
The patent replaces the computationally expensive linear programming approach with a more efficient heuristic algorithm that provides near-optimal solutions at a fraction of the computational cost. The heuristic method uses simplified rules and approximations to identify the k-vector, sacrificing minimal optimality for significant gains in computational efficiency and scalability, making the approach practical for large-scale document sanitization tasks.
Data Source
AI summary
A system, method, and computer program are provided for text sanitization. The system builds a corpus of document vectors (including tokenizing each document, creating a vector representation based on the tokens, and building a corpus of vector representations), obtains a new document for text sanitization, tokenizes the new document, creates a new document vector based on the tokens in the new document, and accesses the corpus of document vectors. The system filters each of the tokens in the new document against a privacy threshold. The system performs a k-anonymity sanitization process such that the new document vector becomes indistinguishable from at least k other document vectors in the corpus of document vectors. The system replaces or redacts the tokens in the document flagged as unsafe. The system updates the corpus of document vectors to include the new document vector in its form prior to the filtering and k-anonymity sanitization steps.