Horizontally-Scalable Data De-Identification via Parallel Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data anonymization methods, such as k-Anonymity, face challenges in efficiently processing large datasets due to their NP-hard nature and high computational complexity, especially in distributed systems where data transfer between processors is costly, leading to slowed computations and increased resource usage.
Innovation Solution
The implementation of a cloud computing platform that employs heuristics for organizing the search space of full domain generalization, pruning unnecessary generalizations, and approximating equivalence class sizes and suppression components to reduce dependency on data transfer between processors, allowing for concurrent computation and efficient determination of candidate generalizations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If k-Anonymity method is used for data anonymization, then data privacy protection is improved, but computational complexity increases and processing speed decreases
Solution Approach 1:
The patent divides the large dataset into multiple partitions that can be processed independently and in parallel across different processors. Each partition is anonymized separately using k-Anonymity, then the results are merged. This segmentation reduces the computational burden on individual processors and enables concurrent execution, thereby improving processing speed while maintaining privacy protection.
Solution Approach 2:
The patent performs preliminary actions by pre-computing equivalence classes and generalization hierarchies before the actual anonymization process. Candidate generalizations are identified and prepared in advance, reducing the computational complexity during the main anonymization phase. This preliminary organization of data structures speeds up the overall processing without compromising the k-Anonymity guarantee.
2Manufacturing precision
If full domain generalization is performed to ensure k-Anonymity, then data anonymization accuracy is improved, but computational resources and time consumption increase
Solution Approach 1:
The patent extracts and focuses only on the essential generalizations that are necessary to achieve k-Anonymity, rather than performing exhaustive full domain generalization. By identifying and removing unnecessary generalization steps, the system maintains anonymization accuracy while significantly reducing computational time and resource consumption.
Solution Approach 2:
The patent applies partial generalization where sufficient but not excessive transformation is performed to meet k-Anonymity requirements. Instead of applying full domain generalization to all attributes, the system selectively generalizes only those attributes and to those extents necessary to achieve the required anonymity level, thereby reducing computation time while maintaining adequate anonymization accuracy.
3Productivity
If data is processed in distributed systems for parallel computation, then processing speed is improved, but data transfer overhead between processors increases
Solution Approach 1:
The patent segments the dataset into multiple partitions that are distributed across different processors. Each processor independently anonymizes its local partition using pre-computed generalization structures, minimizing the need for inter-processor data transfer. This segmentation enables parallel processing while reducing communication overhead between nodes in the distributed system.
Solution Approach 2:
The patent introduces an intermediary layer of pre-computed generalization structures and equivalence class information that mediates between the distributed processors. Instead of processors needing to exchange large amounts of raw data, they communicate only with this intermediary structure, significantly reducing data transfer overhead while maintaining the ability to perform accurate k-Anonymity anonymization across the distributed system.
Data Source
AI summary
A method receives data from a data source. The method generates a plurality of generalizations of the data. The method sends the plurality of generalizations of the data to a plurality of execution nodes, wherein each of the plurality of execution nodes includes computational resources to compute a candidate generalization using an information loss scoring function. The method receives a candidate generalization from each of the plurality of execution nodes. The method selects a preferred generalization from the plurality of candidate generalizations. The method generates an anonymized view of the data set using the preferred generalization.