Graph Database Data Quality Detection via Subgraph Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Graph databases face challenges in detecting data quality issues due to the complexity of relationships between nodes and attributes, especially in large datasets, where inconsistencies in node relations and attribute values can lead to inaccuracies and duplicate records, making it difficult to efficiently identify and resolve data quality problems.
Innovation Solution
A method is introduced to identify data quality issues by analyzing subgraphs within a graph database, using data quality rules to evaluate entity and edge attribute values, and applying association algorithms to detect inconsistencies, allowing for the marking of potential issues and subsequent processing to improve data reliability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If data quality rules are applied to the entire graph database to detect inconsistencies, then measurement precision of data quality issues is improved, but loss of time increases due to the large size and complexity of the graph
Solution Approach 1:
The patent divides the graph database into multiple partitions, where each partition contains a subset of nodes and edges. Data quality rules are applied independently to each partition rather than the entire graph, significantly reducing the computational time and resources required while maintaining detection accuracy within each partition.
Solution Approach 2:
The patent performs preliminary indexing and organization of graph data before applying quality rules. Metadata about node and edge relationships is pre-processed and stored in optimized structures, enabling faster evaluation of data quality constraints without repeatedly analyzing the entire graph structure during detection.
2Reliability
If complex data quality rules are applied to evaluate all node and edge relationships, then reliability of data quality detection is improved, but device complexity increases
Solution Approach 1:
The system separates complex data quality rules into modular, independent rule components that can be evaluated separately on different graph partitions. This modularization reduces the complexity of any single processing unit while maintaining the comprehensive coverage of all quality constraints across the entire database.
Solution Approach 2:
The patent introduces an intermediary layer of rule compilation and optimization that translates complex data quality constraints into efficient evaluation procedures. This intermediary processing stage simplifies the actual detection operations by pre-computing rule implications and organizing them for rapid evaluation against graph data.
3Measurement precision
If the entire graph is scanned to detect data quality issues, then measurement precision is improved, but productivity decreases due to the large volume of data
Solution Approach 1:
The graph database is divided into multiple manageable partitions, allowing parallel processing of data quality detection across partitions. This segmentation enables the system to maintain comprehensive detection coverage while significantly improving processing throughput and productivity through concurrent evaluation of multiple partitions.
Solution Approach 2:
The patent applies data quality rules selectively to relevant portions of the graph based on query parameters and detected anomaly patterns. Rather than uniformly processing the entire graph, the system focuses computational resources on high-risk areas and partitions, achieving sufficient detection precision with reduced overall processing requirements.
Data Source
AI summary
Methods, computer program products and/or systems are provided that perform the following operations: obtaining a first graph comprising first nodes representing first entities and first edges representing relationships between first entities, the first nodes being associated with first entity attributes descriptive of the first entities represented by the first nodes, the first edges being associated with first edge attributes descriptive of the relationships represented by the first edges; determining a first subgraph for a certain node of the first nodes of the first graph, the first subgraph including the certain node and at least one neighboring node of the certain node; and determining a data quality issue regarding the certain node based, at least in part, on applying one or more applicable rules of a set of data quality rules to first entity attribute values and first edge attribute values of the first subgraph.