Random Walks for Dissimilar Record Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for identifying dissimilar records in a dataset are either computationally expensive or unreliable, failing to effectively capture the diversity needed for unbiased algorithm training and data management.
Innovation Solution
A graph-based approach using random walks to score and identify dissimilar nodes in a dataset, ensuring a diverse set of records for training data that reduces bias and improves ethical parameters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If explicit computation methods are used to identify similar nodes, then measurement precision is improved, but productivity deteriorates due to high computational cost
Solution Approach 1:
The patent employs random walk methods as computationally inexpensive sampling techniques to estimate node similarity, replacing expensive explicit computation. Multiple random walks are performed and aggregated to achieve reliable results without the high cost of exhaustive computation, effectively using cheap computational operations to approximate the results of expensive exact methods.
2Measurement precision
If graph-based methods are used to find dissimilar nodes, then measurement precision is improved, but productivity deteriorates due to computational expense
Solution Approach 1:
The patent performs a limited number of random walks from each node rather than computing all pairwise distances in the graph. This partial action approach explores only a subset of the graph structure sufficient to identify dissimilar nodes, avoiding the computational burden of exhaustive graph traversal while maintaining detection accuracy.
3Productivity
If non-graph metrics are applied to identify dissimilar nodes, then productivity is improved by reducing computational cost, but measurement precision deteriorates by ignoring graph relationships
Solution Approach 1:
The patent uses random walks as an intermediary mechanism that bridges the gap between non-graph metrics and full graph analysis. The random walks incorporate graph structure information indirectly by sampling paths through the graph, providing a computationally efficient method that still captures graph relationships without requiring exhaustive computation of all graph properties.
4Productivity
If clustering algorithms are used to find dissimilar records, then productivity is improved, but measurement precision deteriorates because clustering optimizes for similarity not dissimilarity
Solution Approach 1:
Instead of using clustering algorithms that optimize for similarity and then selecting representatives, the patent inverts the approach by directly optimizing for dissimilarity identification through random walk sampling. The random walks are specifically designed to explore graph structure and identify nodes that are dissimilar to a given node, providing direct rather than indirect dissimilarity detection.
Data Source
AI summary
One example method includes identifying dissimilar items in a data set. A data set may be walked one or more times and the nodes or vertices of the data set may be scored based on the number of times the nodes are touched during the walks. Scores below a threshold score are determined to be dissimilar nodes in the data set. This allows a diverse set of nodes to be identified. A dissimilar data set may be used to prevent unintentional bias in algorithmic training.