Data Transformation Apparatus for Efficient Similarity Determination
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for determining similarity between datasets describing relationships between humans or objects, such as communication logs, face challenges in maintaining accuracy and computational efficiency, particularly when dealing with combinations of interrelated items like source hosts, destination hosts, and port numbers.
Innovation Solution
A data transformation method that selects items to divide records into selected and non-selected groups, calculates similarity between these groups, and generates weight datasets to transform the original datasets into similarity-determining datasets, using matrix operations to optimize similarity calculation while maintaining computational efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If an exhaustive approach is taken to determine record-to-record associations to maximize similarity between datasets, then the accuracy of similarity determination is improved, but the computational load explodes making it impossible to achieve the goal within a realistic time frame
Solution Approach 1:
The patent divides the dataset into multiple subsets based on item groups, where each subset contains records sharing common item values. By processing these subsets separately and computing similarity within each subset, the method avoids the combinatorial explosion of exhaustive comparison while maintaining accurate similarity measurement through structured decomposition of the comparison task.
2Productivity
If graph kernels are used to efficiently calculate similarity with small computation, then the computational load is reduced, but some part of the original data is lost leading to degraded accuracy of similarity determination
Solution Approach 1:
The patent applies local quality by computing similarity measures specifically for selected item groups rather than uniformly processing all items. Each item group is assigned a weight reflecting its importance, and similarity is calculated locally within groups before being aggregated. This selective, localized approach maintains accuracy for critical attributes while reducing overall computational burden.
Solution Approach 2:
The patent transforms the original data by applying weight matrices to item groups, changing the parameter representation from raw values to weighted similarity scores. This parameter transformation enables efficient computation by converting the similarity determination problem into a matrix operation that can be solved without exhaustive enumeration, while preserving the essential information through the weighted aggregation.
Data Source
AI summary
A data transformation apparatus selects items one by one and generates a first weight dataset and a second weight dataset on the basis of similarity between first records in a first dataset and second records in a second datasets. The first records and second records respectively include first item values and second item values that belong to the selected item. Based on the first weight dataset, the data transformation apparatus transforms the first dataset into a first similarity-determining dataset including third records. Each third record includes a numerical value that indicates a relationship between transformed item values belonging to different items. Further, based on the second weight dataset, the data transformation apparatus transforms the second dataset into a second similarity-determining dataset including fourth records. Each fourth record includes a numerical value that indicates a relationship between transformed item values belonging to different items.