Data Profiling via Location Information Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data profiling methods struggle to efficiently characterize values in specified fields of datasets, particularly when relationships and dependencies between fields are unknown, and when computing data quality measures for combinations of fields.
Innovation Solution
A method that involves accessing collections of records from data storage systems, generating location information for distinct values in specified fields, and processing this information to create lists of entries that characterize values and their locations, enabling efficient computation of data quality measures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional data profiling methods are used to examine source datasets, then data quality measures can be obtained, but the process is inefficient when characterizing values in specified fields and computing data quality measures for combinations of fields
Solution Approach 1:
The patent pre-computes and stores location information for each distinct value in each field before actual data profiling queries are executed. This preliminary action creates a ready-to-use index structure that enables rapid retrieval during profiling operations, eliminating the need to scan entire datasets when computing data quality measures for field combinations
Solution Approach 2:
The patent creates a compressed representation (copy) of the data dataset that includes location information for each distinct value. This copy stores essential profiling data in a compact format that can be quickly accessed and manipulated without dealing with the full original dataset, significantly improving computation speed for data quality measures
2Productivity
If location information is generated for every distinct value in specified fields, then efficient computation of data quality measures is enabled, but the amount of stored information increases
Solution Approach 1:
The patent applies local quality by storing location information selectively for each field based on its specific characteristics and the profiling needs of that field. Rather than uniformly storing all possible data, the system tailors the location information storage to each field's requirements, optimizing the balance between query efficiency and storage usage
Solution Approach 2:
The patent transforms the raw data into a different parameter representation by storing location information as distinct value indices rather than actual data values. This parameter change compresses the storage requirements while preserving the ability to efficiently compute data quality measures, as location information can be derived from the compressed representation
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
Profiling data includes processing an accessed collection of records (203), including: generating, for a first set of distinct values appearing in a first set of one or more fields, corresponding location information; generating, for the first set of fields, a corresponding list of entries (209) identifying a distinct value from the first set of distinct values and the location information for the distinct value; generating, for a second set of one or more fields, a corresponding list of entries (209), with each entry identifying a distinct value from a second set of distinct values appearing in the second set of fields; and generating result information (240), based at least in part on: locating at least one record of the collection using the location information for at least one value appearing in the first set of fields, and determining at least one value appearing in the second set of fields of the located record.