Data Standardization Using Self-Generated Reference Terms
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Automated data standardization is challenging due to the lack of available reference data and the high overhead in preparing such data, especially for large datasets, as existing methods struggle to accurately handle inconsistencies like abbreviations and misspellings without relying on external references.
Innovation Solution
The techniques involve determining standard representation terms within the data itself, either by clustering similar terms or using a combination of the data and external reference data, to identify approximate matches and select standard representation terms based on frequency and similarity, potentially with user input for accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If automated data standardization is performed using reference data, then standardization accuracy is improved, but the overhead in preparing reference data increases significantly
Solution Approach 1:
The system performs self-service by automatically generating standard representation terms from the input data itself through clustering similar terms, eliminating the need for external reference data preparation while maintaining standardization accuracy
Solution Approach 2:
The system performs preliminary clustering and analysis of the input data to identify standard representation terms before the actual standardization process, reducing the need for extensive reference data preparation later
2Measurement precision
If manual data standardization is performed, then standardization accuracy is improved, but scalability deteriorates for large datasets
Solution Approach 1:
The system replaces manual mechanical standardization processes with automated computational clustering algorithms that analyze term similarities and frequencies to identify standard representations, achieving both accuracy and scalability
Solution Approach 2:
The system creates standardized representations by copying and generalizing patterns from the input data itself, using clustered term patterns as templates for standardization without requiring manual creation of reference data
3Measurement precision
If external reference data is used for standardization, then standardization accuracy is improved, but device complexity increases
Solution Approach 1:
The system extracts standard representation patterns directly from the input data by clustering similar terms and identifying frequent patterns, eliminating the need for external reference data and reducing system complexity
Solution Approach 2:
The clustering-based approach serves multiple functions: it identifies standard representations, groups variant terms, and adapts to different data types without requiring separate reference data for each type, reducing overall system complexity
Data Source
AI summary
Techniques are disclosed for standardization of data. According to a first technique, standard representation terms are determined for to-be-standardized data using the to-be-standardized data itself and without using any external reference data. According to a second technique, a combination of the to-be-standardized data and an external reference is used to determine standard representation terms for the to-be-standardized data.