Big Data K-Anonymization via Parallel Semantic Micro-Aggregation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for achieving k-anonymity in big data, such as micro-aggregation, are slow due to the need for numerous iterations and are inadequate for processing nominal attributes, which are essential in big data, as they do not consider semantic similarity.
Innovation Solution
A method that determines aggregation clusters by computing an average record and finding the most distant records using a distance measure, with a predefined repetition counter to reduce iterations, and applies hierarchical generalization for nominal attributes to improve processing speed and accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If micro-aggregation method is used to achieve k-anonymity, then privacy protection is improved, but processing speed deteriorates due to huge number of iterations required
Solution Approach 1:
The patent segments the large dataset into multiple smaller blocks or partitions that can be processed independently and in parallel. This segmentation reduces the computational burden on each processing unit, allowing the system to handle big data efficiently while maintaining k-anonymity requirements through distributed micro-aggregation operations.
Solution Approach 2:
The patent performs preliminary actions by pre-computing certain metrics, indexing data structures, and preparing data blocks before the actual k-anonymity processing. This includes pre-calculating distance measures, organizing data into suitable formats, and setting up parallel processing frameworks, which significantly reduces the number of iterations needed during the actual anonymization process.
2Device complexity
If traditional distance measure is used for nominal attributes, then calculation simplicity is improved, but semantic similarity is lost leading to inadequate anonymization
Solution Approach 1:
The patent transforms the measurement parameters for nominal attributes by introducing semantic similarity metrics that go beyond simple equality checks. It employs parameters such as semantic distance, hierarchical category matching, or domain-specific similarity measures that capture the meaningful relationships between nominal values, thereby improving anonymization quality while maintaining computational feasibility.
Solution Approach 2:
The patent introduces intermediary concepts or mapping layers between raw nominal attributes and their semantic representations. This intermediary layer enables the system to compute meaningful distances and similarities for nominal attributes by translating them into a common semantic space, allowing for more accurate clustering and generalization without excessive computational complexity.
3Measurement precision
If complete dataset iteration is performed for each element in each iteration, then clustering accuracy is improved, but processing time increases making it unsuitable for big data
Solution Approach 1:
The patent divides the complete dataset into multiple segments or blocks that can be processed independently. Instead of iterating through the entire dataset for each element in every iteration, the system processes smaller segments in parallel, reducing the computational complexity from O(n²) to approximately O(n²/p) where p is the number of parallel processing units, thereby maintaining clustering accuracy while dramatically reducing processing time.
Solution Approach 2:
The patent performs preliminary actions by pre-computing distance matrices, indexing data points by their feature characteristics, and identifying potential cluster centers before the main iterative process. This preliminary processing reduces the search space and allows for more efficient convergence, maintaining clustering accuracy while reducing the number of full dataset iterations required.
Data Source
AI summary
Provided is a method for anonymizing datasets having sensitive information, including the steps of determining a dataset of records to be assigned to aggregation clusters; computing an average record of the dataset on the basis of a predefined repetition counter; finding a most distant first record to the average record using a distance measure; finding a most distant second record from the first record using the distance measure; forming a first aggregation cluster around the first record and a second aggregation cluster around the second record; and generating a new dataset by subtracting the first cluster and the second cluster from the previous dataset.