Anonymizing User Identifiable Information via Mapping Table
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Companies face inefficiencies and high processor intensity when removing user identifiable information (UII) from non-indexed data storage systems, particularly in Hadoop environments, due to the need to scan and rewrite entire files upon user account deletion, which is time-consuming and unmanageable for large datasets.
Innovation Solution
Implementing anonymization procedures that replace UII with randomly generated information (RGI) and use an anonymization mapping table to disassociate UII from RGI, allowing for efficient deletion of UII without scanning and rewriting entire files, thus overcoming indexing issues in non-indexed storage systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional scanning and rewriting methods are used to remove UII from non-indexed storage systems, then complete UII removal is achieved, but processing time and processor intensity become unmanageably high
Solution Approach 1:
The system performs preliminary anonymization by replacing UII with RGI during data ingestion or batch processing, creating an anonymization mapping table in advance. When a user deletion request occurs, the system only needs to disassociate the UII from its RGI in the mapping table rather than scanning entire files, dramatically reducing processing time while ensuring complete UII removal.
Solution Approach 2:
The invention introduces randomly generated information (RGI) as an intermediary between UII and the actual data. The anonymization mapping table serves as a mediator that links UII to RGI, allowing the system to efficiently manage and remove UII references without directly processing the underlying data files. This intermediary layer enables fast disassociation through simple table updates rather than file rewriting.
2Reliability
If traditional scanning and rewriting methods are used to remove UII from non-indexed storage systems, then complete UII removal is achieved, but processor intensity becomes unmanageably high
Solution Approach 1:
The system performs preliminary anonymization by replacing UII with RGI during data ingestion or batch processing, creating an anonymization mapping table in advance. When a user deletion request occurs, the system only needs to disassociate the UII from its RGI in the mapping table rather than scanning entire files, dramatically reducing processing time while ensuring complete UII removal.
Solution Approach 2:
The invention introduces randomly generated information (RGI) as an intermediary between UII and the actual data. The anonymization mapping table serves as a mediator that links UII to RGI, allowing the system to efficiently manage and remove UII references without directly processing the underlying data files. This intermediary layer enables fast disassociation through simple table updates rather than file rewriting.
3Reliability
If entire files are scanned and rewritten for each user deletion request, then UII removal is ensured, but scalability becomes unmanageable for large datasets
Solution Approach 1:
The system performs preliminary anonymization by replacing UII with RGI during data ingestion or batch processing, creating an anonymization mapping table in advance. When a user deletion request occurs, the system only needs to disassociate the UII from its RGI in the mapping table rather than scanning entire files, dramatically reducing processing time while ensuring complete UII removal.
Solution Approach 2:
The invention segments the UII management process into two independent components: the anonymization mapping table (containing UII-RGI associations) and the actual data files (containing RGI references). This segmentation allows the system to manage UII removal through lightweight table operations independent of file size, enabling scalability to petabyte-scale datasets without increasing processing complexity.
Data Source
AI summary
The disclosed techniques provide systems and methods for anonymizing various portions of information, action logs, end-user information, and/or other data sets that are stored in non-indexed storage systems. More specifically, various anonymization procedures are described for redacting UII and/or replacing UII in raw data with randomly generated information (RGI). The anonymization process is performed on a rolling basis as raw data is received. An anonymization mapping table maps (or associates) the replaced UII in the anonymized data to the RGI, and eventually all raw data can be deleted.