Data Token Clustering for Dataset Error Correction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data processing systems face challenges in correcting errors and inconsistencies in datasets, such as spelling mistakes, missing data, and anomalous values, which affect the accuracy and consistency of results.
Innovation Solution
A computer-implemented method that groups data tokens into clusters based on similarity, selects a reference cluster, transforms tokens in other clusters to match the reference cluster, and updates the dataset with identified corrections, leveraging domain similarity and syntactic measures to propose and apply repairs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If data tokens are grouped into clusters based on similarity and transformations are applied to match reference clusters, then data consistency and accuracy are improved, but the computational complexity and processing time increase
Solution Approach 1:
The patent segments the data correction problem into distinct phases: clustering tokens by similarity, selecting reference clusters, generating transformation candidates, and evaluating corrections. This segmentation allows each sub-problem to be handled independently with appropriate algorithms, improving overall accuracy while managing computational complexity through modular processing.
Solution Approach 2:
The patent performs preliminary clustering and reference selection before actual correction operations. By pre-organizing tokens into clusters and identifying reference clusters in advance, the system reduces the computational burden during the correction phase, as transformations only need to be evaluated against pre-established references rather than all possible token combinations.
2Stability of the object's composition
If clustering and transformation evaluation are performed to correct data tokens, then data consistency is improved, but the processing time increases
Solution Approach 1:
The patent implements feedback mechanisms where transformation candidates are evaluated based on their effect on reference cluster properties. The system uses domain knowledge and syntactic measures to assess whether transformations maintain or improve data consistency, providing feedback that guides the selection of optimal corrections while avoiding unnecessary processing of invalid transformations.
Solution Approach 2:
The patent changes parameters such as similarity thresholds, cluster sizes, and transformation costs to balance processing time and data consistency. By adjusting these parameters based on the specific dataset and correction goals, the system can optimize the trade-off between thorough evaluation and processing efficiency.
Data Source
AI summary
Correcting data in a dataset. A set of data tokens from a tabular data store are grouped into a plurality of different clusters based on similarity of tokens. A reference cluster is selected from among the plurality of different clusters such that the plurality of clusters includes a reference cluster and one or more other clusters. One or more tokens in the one or more other clusters are transformed. The effect on the reference cluster of adding the transformed tokens to the reference cluster is determined. Using this information, a correction for a token in the dataset is identified. The data store is updated to correct the token using the identified correction.