Graph-Based Data Structure Relationship Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large and noisy data sets present challenges in identifying relationships between data elements, leading to issues with data cleanliness, redundancy, and inefficiencies in database management due to errors, variations, and inconsistencies across different databases.
Innovation Solution
An improved data structure processing architecture that uses a graph data structure to detect and flag duplications, normalize data, and establish equivalence relations, enabling the identification of relationships and reducing redundancy through the use of an adjacency matrix and equivalence classes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional data processing methods are used on voluminous data sets, then data processing can be performed, but relationship detection becomes increasingly difficult and inefficient as data volume and velocity grow
Solution Approach 1:
The patent segments the voluminous data set into smaller equivalence classes based on detected relationships. By dividing the large data processing task into manageable equivalence class segments, the system can efficiently detect relationships within each segment while maintaining overall productivity despite the large total data volume.
Solution Approach 2:
The patent introduces an intermediary processing layer that uses graph data structures and equivalence relation detection to mediate between raw voluminous data and final processed results. This intermediary approach transforms the direct relationship detection problem into a multi-step process involving graph construction, equivalence class identification, and relationship inference, making detection feasible at scale.
2Quantity of substance
If data from multiple sources with formatting problems is processed, then more comprehensive data can be obtained, but data cleanliness and consistency deteriorate due to errors, variations, and inconsistencies
Solution Approach 1:
The patent extracts and separates the relationship detection logic from the raw data processing. By using graph data structures to represent relationships independently from the actual data values, the system can handle voluminous multi-source data while maintaining reliability through structured relationship representation that is insensitive to formatting variations and inconsistencies.
Solution Approach 2:
The patent changes the parameter representation from raw data values to graph-based relationship indicators. By transforming data from its original formatted state into graph nodes and edges with equivalence relations, the system preserves the quantity of data from multiple sources while improving reliability through standardized relationship parameters that are invariant to formatting problems.
3Reliability
If equivalence relations are established among data records, then duplications can be identified and removed, but data redundancy increases during the relationship detection process
Solution Approach 1:
The patent performs preliminary relationship detection and equivalence class identification before final data processing. By pre-establishing the graph structure and identifying equivalence relations in advance, the system can then efficiently remove duplications in a single pass, minimizing temporary redundancy rather than allowing it to accumulate during processing.
Solution Approach 2:
The patent uses graph data structures as a copy representation of the relationship information rather than duplicating the actual data records. The graph captures equivalence relations as structural information, allowing the system to identify and remove duplications based on relationship patterns without creating excessive redundant copies of the original voluminous data.
Data Source
AI summary
A computer implemented system and method for automated estimation of relationships among a plurality of data elements. The approach includes processing elements of one or more data sets to establish linkage relations among the data records, and then extending the linkage relations based on one or more equivalence relations, stored as linkage data structures. The generated data structures are used for computationally simplifying the data sets by consolidating data records or removing redundancies, such as duplicates, and may be used to yield a compressed data representation or data structure.