Dataset Association Search Optimization via Size-Based Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for determining data elements with association relations in two datasets are inefficient, especially when dealing with large datasets, as they involve high time overhead due to direct searching and traversing operations without comparing the number of data elements.
Innovation Solution
A method that compares the number of data elements in two datasets, selecting one as a search dataset and the other as a traverse dataset based on the comparison, and uses an index to efficiently find candidate data elements with association relations, optimizing the search and traverse operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If direct searching and traversing operations are performed on two datasets without comparing their sizes, then data element association can be determined, but time overhead and computational complexity increase significantly
Solution Approach 1:
The patent applies preliminary action by comparing the sizes of the two datasets before performing search operations. This preliminary size comparison allows the system to identify which dataset is smaller and use it as the search dataset, thereby optimizing the subsequent search process and reducing time overhead for determining data element associations.
Solution Approach 2:
The patent inverts the conventional approach by selecting the smaller dataset as the search dataset rather than the larger one. This inversion principle optimizes the search process by reducing the number of comparisons needed, as searching through the smaller dataset requires fewer operations than searching through the larger dataset.
2Reliability
If the larger dataset is used as the search dataset, then all data elements can be searched, but the computational complexity and time overhead increase
Solution Approach 1:
The patent applies the inversion principle by selecting the smaller dataset as the search dataset instead of the larger one. This approach maintains reliability in determining data element associations while significantly reducing computational complexity, as the search operation on the smaller dataset requires fewer comparisons and less processing power.
Solution Approach 2:
The patent changes the parameter of dataset selection from size-based (using larger dataset) to complexity-based (using smaller dataset). By changing this selection parameter, the system achieves the same association determination reliability with lower computational complexity and reduced time overhead.
Data Source
AI summary
The present disclosure relates to a method, apparatus and computer program product for managing datasets. According to example implementations of the present disclosure, there is provided a method for managing datasets. In the method, a first dataset comprising a first number of data elements and a second dataset comprising a second number of data elements may be obtained. The first number may be compared with the second number. One of the first dataset and the second dataset may be selected as a search dataset and the other may be selected as a traverse dataset based on a comparison result. The search dataset may be searched for a candidate data element, here the candidate data element has an association relation with a data element comprised in the traverse dataset. Further, there is provided an apparatus and computer program product for managing datasets.