Database Safety Assessment Using Kd-Tree Neighbor Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for assessing the safety of large-scale databases face significant processing time challenges as the number of records increases, with record linkage processing time growing quadratically, making it impractical for large datasets.
Innovation Solution
Implementing a safety assessment apparatus and method that uses a neighbor search with a tree structure or hashing to efficiently calculate re-identification rates by reducing the number of distance calculations to O(N log N), allowing for practical processing times by using a kd-tree and excluding duplicate records to focus on a predetermined neighborhood number.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If record linkage is performed by calculating distances between all records in the original database and target records in the secure database, then re-identification accuracy is improved, but processing time increases quadratically with the number of records
Solution Approach 1:
The original database is divided into multiple clusters using k-means clustering, where each cluster contains a subset of records. During record linkage, only records within the same cluster are compared, reducing the number of distance calculations from O(N^2) to approximately O(N^2/k) where k is the number of clusters. This segmentation maintains re-identification accuracy by keeping related records together while dramatically reducing processing time.
Solution Approach 2:
Kd-trees are constructed in advance to organize the spatial distribution of records in the database. This preliminary action creates an efficient search structure that allows for fast nearest-neighbor queries during record linkage, reducing the time complexity from linear search O(N) per query to O(log N) per query, thereby solving the quadratic time problem while maintaining accurate re-identification.
2Productivity
If kd-trees are used to apply record linkage for large datasets, then processing time is reduced, but implementation complexity increases
Solution Approach 1:
The patent combines multiple techniques into a unified framework: k-means clustering for initial data organization, kd-tree construction for efficient spatial indexing, and distance-based record linkage for re-identification assessment. This merging of techniques creates a cohesive system that achieves O(N log N) processing efficiency while managing implementation complexity through integrated design.
Solution Approach 2:
The patent introduces intermediate structures (clusters and kd-trees) as mediators between the raw data and the final record linkage process. These intermediaries organize data in advance, allowing the main record linkage algorithm to operate efficiently without directly handling all N^2 comparisons, thus improving productivity while containing complexity within the intermediate layers.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The safety of a database that conceals large-scale data is efficiently assessed. A database storage (10) stores an original database and a secure database. A neighbor record search part (11) obtains a neighbor record set by a neighbor search of the original database for each record in the secure database. A nearest neighbor record calculation part (13) calculates a distance between each record in the secure database and each record in the neighbor record set and obtains a nearest neighbor record. A re-identification determination part (14) calculates a re-identification rate of a record based on whether the record in the original database corresponding to each record in the secure database matches the nearest neighbor record. A re-identification rate calculation part (15) calculates a re-identification rate of the secure database based on the re-identification rate calculated for each record in the secure database.