Obfuscating Personal Identifiers in Distributed Event Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge lies in efficiently processing and removing personal identifiers from large volumes of telemetric data in a loosely-coupled distributed system, where personal data is scattered across multiple storage locations, making it complex to track and delete user information while maintaining high throughput.
Innovation Solution
A real-time event processing system uses a cryptographic hash function to obfuscate personal identifiers, creating reversible maps and delete tables to distinguish between active and deleted identifiers, allowing for efficient deletion and data analysis without referencing personal information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If personal identifiers are stored in a reversible map for later retrieval, then user identity can be recovered when needed, but the system must maintain additional data structures and processing overhead
Solution Approach 1:
The patent extracts the personal identifier from the event data and stores it separately in a reversible map, while only the obfuscated version remains in the event. This separation allows the system to maintain identity recovery capability without burdening the main event processing pipeline.
Solution Approach 2:
The reversible map acts as an intermediary data structure that bridges the gap between obfuscated event data and original personal identifiers. It provides a controlled mechanism for identity recovery without exposing the full identifier in the main data flow.
2Object-affected harmful factors
If personal identifiers are deleted from storage, then user privacy is protected, but the system loses the ability to retrieve user identity when needed
Solution Approach 1:
The personal identifier is extracted from the main event data and placed in a separate deleteable storage location. When deletion is requested, only the extracted copy is removed while the obfuscated version in the event remains intact, preserving both privacy and analytical capability.
Solution Approach 2:
The system creates an obfuscated copy of the personal identifier that can persist in event data without compromising privacy. This copy contains sufficient information for analysis but cannot be reversed without the original identifier, which can be selectively deleted.
3Object-affected harmful factors
If all storage locations are searched to find personal identifiers for deletion, then complete privacy protection is achieved, but processing time and computational resources are excessively consumed
Solution Approach 1:
The personal identifier is extracted and stored in a dedicated, easily accessible location separate from the main event data. This allows the system to delete identifiers by accessing this specific location rather than searching through all possible storage locations in the distributed system.
Solution Approach 2:
The system performs preliminary extraction of personal identifiers to a centralized location before deletion operations. This preparation step enables efficient batch deletion without requiring subsequent searches through distributed storage locations.
4Object-affected harmful factors
If obfuscation methods are applied to personal identifiers, then privacy is protected during data analysis, but computational overhead increases
Solution Approach 1:
The system uses simple obfuscation techniques that create pseudonymous identifiers which are sufficient for analysis purposes. These obfuscated values are computationally inexpensive to generate and process, providing adequate privacy protection without heavy computational costs.
Data Source
Figure 1
Figure 2A~2C
Figure 2D~2E
AI summary
A real-time event processing system receives event data containing telemetric data and one or more personal identifiers. The personal identifier in the event data is replaced with an obfuscated value so that the telemetric data may be used without reference to the personal identifier. A reversible map is used to reverse the obfuscated personal identifier to its original value. In the case when a request is received to delete the mapped personal identifier, the link to the entry in the reversible map is broken by associating the personal identifier with a different obfuscated value.