Metadata Deduplication Using Geohash and Word Embeddings
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in accurately merging metadata profiles for places from various sources due to conflicting and noisy data, leading to inaccurate geospatial coordinates and metadata attributes, which affects business decision-making.
Innovation Solution
The implementation of a deduplication system using machine learning-based models, such as recurrent neural networks and extreme gradient boosted trees, to compare and merge metadata profiles, hyper-locate geospatial coordinates within building structures, and define physical boundaries, resulting in more accurate and reliable location-specific data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If metadata profiles are gathered from multiple sources, then the quantity of location-specific data is increased, but the accuracy and reliability of the data deteriorates due to conflicting and noisy information
Solution Approach 1:
The patent extracts and removes duplicate and conflicting metadata entries from multiple sources through a deduplication system. The system identifies redundant place entries across different metadata sources and eliminates duplicates by comparing place identifiers, names, and geospatial coordinates, retaining only the most accurate and reliable information from each source.
Solution Approach 2:
The patent introduces a machine learning-based deduplication system as an intermediary between multiple metadata sources and the final place library. This intermediary system processes, compares, and reconciles conflicting information from various sources, using trained models to determine the most accurate metadata and geospatial coordinates for each place.
2Measurement precision
If place names from multiple sources are compared using string similarity, then duplicate places can be identified, but false positives occur when different places have similar names
Solution Approach 1:
The patent transforms place name comparison from simple string similarity to a multi-parameter analysis including place identifiers, normalized names, categories, and geospatial coordinates. The system uses parameter changes such as name normalization (removing stop words, converting to lowercase) and coordinate-based validation to distinguish between truly duplicate places and different places with similar names.
Solution Approach 2:
The patent adds additional dimensions to the duplicate detection process by incorporating geospatial coordinates, place categories, and multiple metadata attributes beyond just name similarity. This multi-dimensional approach allows the system to differentiate between places with similar names by examining their spatial locations and other distinguishing characteristics.
3Area of stationary object
If geospatial coordinates from multiple sources are merged, then the coverage of location data is improved, but the precision of centroids deteriorates due to conflicting coordinate information
Solution Approach 1:
The patent implements a feedback mechanism where the deduplication system uses geospatial coordinates to validate and refine place centroids. The system compares coordinates from multiple sources, identifies outliers and conflicts, and uses the most reliable coordinate sets to establish accurate centroids. The feedback loop continues until consistent and precise geospatial information is achieved for each place.
4Measurement precision
If machine learning models are used to deduplicate metadata profiles, then the accuracy of place identification is improved, but the complexity of the system increases
Solution Approach 1:
The patent applies preliminary action by pre-training machine learning models on large datasets of place metadata before deployment. The models are trained in advance to recognize patterns, similarities, and distinguishing features of place names and attributes. This preliminary training reduces the computational complexity during actual deduplication operations, as the models have already learned the complex relationships and can make faster, more accurate comparisons.
Data Source
AI summary
The technology disclosed relates to deduplicating metadata about places. A feature generator module is configured to generate features for metadata profiles. The metadata profiles represent a plurality of places. The features are based on geohash strings and word embeddings generated for the metadata profiles. A diff generator module is configured to generate diff vectors that pair-wise encode results of comparison between features of paired metadata profiles. A classification module is configured to generate similarity scores for the paired metadata profiles based on the diff vectors. A particular similarity score indicates whether metadata profiles in a particular pair of metadata profiles represent a same place.