Entity Resolution for Aggregation Query Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing business intelligence systems face challenges in performing accurate aggregation queries over databases with inconsistent dimensional data, leading to erroneous results due to unresolved entities appearing with different attributes in various systems, which is exacerbated by data inconsistencies and the dynamic nature of entity resolution.
Innovation Solution
An entity-based business intelligence query process that collects inconsistent database information, runs entity discovery and resolution processes to produce resolved tables, and performs aggregation queries with certainty status indicators to handle inconsistencies, providing guaranteed or non-guaranteed results based on data availability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If entity resolution is performed statically on a data warehouse, then the resolution process can be completed in advance, but the process is very time-consuming and seldom exact, requiring additional corrections as more instances become available
Solution Approach 1:
The patent transforms static entity resolution into a dynamic process by implementing incremental entity resolution that continuously updates resolution results as new data instances become available. The system dynamically discovers entities and updates their attributes in real-time, allowing the resolution to adapt and improve accuracy over time without requiring complete re-processing of the entire data warehouse.
Solution Approach 2:
The patent performs preliminary entity discovery and resolution on incoming data streams before they are fully integrated into the data warehouse. By pre-processing and resolving entities incrementally as data arrives, the system reduces the time and computational resources required for final entity resolution while improving accuracy through continuous refinement.
2Productivity
If aggregation queries are performed directly on unresolved databases, then the query processing is simple and fast, but the results are erroneous due to data inconsistency
Solution Approach 1:
The patent introduces an intermediary entity resolution layer that sits between the unresolved database and the aggregation query processing. This intermediary dynamically resolves entities and their attributes at query time, allowing the system to maintain fast query processing while ensuring accurate aggregation results by resolving data inconsistencies before aggregation occurs.
Solution Approach 2:
The patent implements dynamic entity resolution that adapts to each query by resolving entities based on the specific query requirements and available data. This dynamic approach allows the system to maintain high query processing speed while ensuring accuracy by resolving only the necessary entities for each query rather than pre-resolving all entities.
3Measurement precision
If entity discovery and resolution is performed to resolve data inconsistencies, then the accuracy of aggregation results improves, but the device complexity and processing time increase
Solution Approach 1:
The patent segments the entity resolution process into distinct modular components including entity discovery, entity matching, attribute resolution, and query integration. Each component handles a specific aspect of the resolution process independently, reducing overall system complexity while maintaining high accuracy through coordinated operation of these specialized modules.
Solution Approach 2:
The patent implements self-service entity resolution where the system automatically discovers entities, resolves their attributes, and integrates them into query processing without requiring manual intervention. The system autonomously manages the complexity of entity resolution by implementing self-learning algorithms that automatically adapt to new data patterns and resolve inconsistencies.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method is disclosed for conducting a query to transform data in a pre-existing database, the method comprising: collecting database information from the pre-existing database, the database information including inconsistent dimensional tables and fact tables; running an entity discovery process on the inconsistent dimensional tables and the fact tables to produce entity mapping tables; using the entity mapping tables to resolve the inconsistent dimensional tables into resolved dimensional tables; and running the query on a resolved database to obtain a query result, the resolved database including the resolved dimensional table.