Text De-identification via NLP Annotator Feedback
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional text de-identification methods face challenges in effectively removing privacy-sensitive personal information while preserving data utility, as they are susceptible to grammatical errors, dependent on large training datasets, and often result in over-scrubbing, failing to recognize all personal information and maintaining data utility.
Innovation Solution
A method and system utilizing natural language processing (NLP) annotators that apply both rule-based and machine learning approaches to identify and anonymize privacy-sensitive information, reinstating innocuous terms to prevent over-scrubbing and ensure comprehensive de-identification, while using syntactic anonymization to preserve data utility.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional text de-identification methods are used to remove privacy-sensitive information, then personal information is removed, but data utility is lost due to over-scrubbing
Solution Approach 1:
The patent applies different processing treatments to different parts of the text based on their sensitivity level. NLP annotators identify specific tokens as direct identifiers, quasi-identifiers, or innocuous terms, and only direct identifiers are removed while quasi-identifiers and innocuous terms are preserved. This localized approach ensures that only the necessary portions are de-identified while maintaining data utility.
Solution Approach 2:
The system performs preliminary classification of text tokens using NLP annotators before applying de-identification. By pre-identifying which tokens are direct identifiers, quasi-identifiers, or innocuous terms, the system can apply appropriate treatments in advance, preventing over-scrubbing and preserving data utility while ensuring effective de-identification.
2Ease of manufacture
If rule-based methods are used for de-identification, then processing is simple, but grammatical errors and incomplete recognition occur
Solution Approach 1:
The patent combines multiple NLP annotators with different strengths (e.g., spaCy for tokenization and POS tagging, NLTK for named entity recognition, and custom regex patterns) to create a composite de-identification system. This composite approach leverages the strengths of each annotator to achieve comprehensive and accurate recognition of direct identifiers, quasi-identifiers, and innocuous terms, overcoming the limitations of individual rule-based methods.
3Measurement precision
If machine learning approaches are used for de-identification, then recognition accuracy improves, but large training datasets are required
Solution Approach 1:
The system uses pre-trained NLP annotators that have already been trained on large datasets by their developers. These annotators (spaCy, NLTK, etc.) come with built-in knowledge of language patterns, grammar, and common entity types. The de-identification system leverages this pre-existing training without requiring additional training data, achieving high recognition accuracy while minimizing training data requirements.
4Reliability
If infrequent terms are filtered out to protect privacy, then re-identification risk decreases, but data utility is reduced
Solution Approach 1:
The system employs a feedback mechanism where NLP annotators continuously analyze the text and identify innocuous infrequent terms that should be preserved. The annotators provide feedback on term context and meaning, allowing the system to distinguish between sensitive infrequent terms that require filtering and non-sensitive infrequent terms that can be retained, thereby maintaining data utility while protecting privacy.
Data Source
AI summary
One embodiment of the invention provides a method for utility-preserving text de-identification. The method comprises generating corresponding processed text for each text document by applying at least one natural language processor (NLP) annotator to the text document to recognize and tag privacy-sensitive personal information corresponding to an individual, and replacing some words in the text document with some replacement values. The method further comprises determining infrequent terms occurring across all processed texts, filtering out the infrequent terms from the processed texts, and selectively reinstating to the processed texts at least one of the infrequent terms that is innocuous. The method further comprises generating a corresponding de-identified text document for each processed text by anonymizing privacy-sensitive personal information corresponding to an individual in the processed text to an extent that preserves data utility of the processed text and conceals the individual's personal identity.