Column Semantic-Type Identification via Fingerprint Similarity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for identifying the semantic-type of a column in a database are resource-intensive and lack accuracy, particularly when determining the semantic-type based on a small sample of values, as they often rely on regular expressions or look-up tables that require scanning the entire column.
Innovation Solution
Generating fingerprints based on column values and comparing them to predefined semantic-type fingerprints to determine the semantic-type, allowing for resource-efficient scanning and improved accuracy by using techniques like GCD, LCD, and RCD fingerprint generation, which produce similarity measures to identify the most likely semantic-type.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If regular expressions or look-up tables are used to identify semantic-type by scanning the entire column, then measurement precision is improved, but use of energy and productivity deteriorate
Solution Approach 1:
The patent segments the column data into fixed-size batches (e.g., 100 rows at a time) and processes each batch independently to generate fingerprints. This allows the system to work with manageable data portions rather than scanning the entire column at once, reducing memory usage and CPU resource consumption while maintaining identification accuracy through cumulative processing of multiple batches.
Solution Approach 2:
The patent uses a limited sample of column values (a small number of batches) to generate fingerprints for semantic-type identification, rather than processing the entire column. This partial action approach provides sufficient accuracy for identification while dramatically reducing resource consumption compared to exhaustive scanning of all column data.
2Use of energy by moving object
If a small sample of column values is used to reduce resource consumption, then use of energy is improved, but measurement precision deteriorates
Solution Approach 1:
The patent processes column data in continuous batches rather than using a single static sample. By accumulating fingerprints from multiple consecutive batches and combining their results, the system achieves higher measurement precision than a small single sample would provide, while still consuming far fewer resources than processing the entire column. The continuous processing of batches allows progressive refinement of the semantic-type identification.
3Measurement precision
If the entire column is scanned to ensure accurate semantic-type identification, then measurement precision is improved, but productivity deteriorates
Solution Approach 1:
The patent divides the column scanning task into segments (batches of fixed row counts) and processes them sequentially. This segmentation enables the system to identify semantic-types after processing only a limited number of batches rather than requiring complete column scanning, thereby improving productivity while maintaining sufficient accuracy through the cumulative fingerprint information from multiple segments.
Solution Approach 2:
The patent performs partial scanning of the column by processing a predetermined limited number of batches (e.g., 10 batches of 100 rows each) rather than scanning the entire column. This partial action provides sufficiently accurate semantic-type identification for practical purposes while dramatically improving productivity by avoiding the need to process all column data.
Data Source
AI summary
Techniques are provided for determining the semantic-type of a target column based on “fingerprints” that are created based on the values in the target column. The fingerprint set for the target column is only generated once, not once per semantic-type. Thus, the target column only needs to be scanned once, and resource usage is minimized. Once generated, the fingerprint set of the column is compared against fingerprint set that corresponds to each semantic-type to generate a “similarity measure”. The semantic-type whose fingerprint set produces the highest similarity measure relative to the target column's fingerprint set is determined to be the semantic-type of the target column.