Data Profiling System Delaying Canonicalization for Distinct Values
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data profiling processes, particularly canonicalization and type-dependent processing, often result in significant delays when dealing with large datasets, making it inconvenient for users developing data processing programs to obtain profiling results in a timely manner.
Innovation Solution
Implementing a method that delays canonicalization and type-dependent processing until after aggregating data values, focusing on distinct values only, which reduces redundant operations and speeds up the process by managing data type information efficiently, allowing data profiling to be performed within development user interfaces with reduced delays.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If canonicalization and type-dependent processing are performed immediately on all data values, then data profiling accuracy is ensured, but processing time increases significantly
Solution Approach 1:
The patent performs preliminary aggregation of data values to identify distinct values before performing canonicalization and type-dependent processing. This preliminary action reduces the number of values that require expensive type-dependent processing, thereby reducing overall processing time while maintaining profiling accuracy on the distinct values that matter most.
Solution Approach 2:
The patent segments the data processing workflow into distinct phases: first aggregating all data values to identify distinct values, then performing type-dependent processing only on those distinct values. This segmentation allows the system to avoid redundant processing of duplicate values while ensuring accurate profiling of all unique data types present in the dataset.
2Loss of information
If data profiling is performed on large datasets with comprehensive processing, then complete profile information is obtained, but the delay becomes unacceptable for development interfaces
Solution Approach 1:
The system performs preliminary aggregation to identify distinct values before executing type-dependent processing. This preliminary step enables the system to minimize the scope of expensive processing operations while still obtaining complete profile information for all unique values, making the overall process fast enough for interactive development interfaces.
Solution Approach 2:
The patent applies partial action by performing type-dependent processing only on distinct values rather than all data values. This partial processing approach provides sufficient profile information for development purposes without the excessive processing time that would result from analyzing every single value occurrence, thereby maintaining interface responsiveness.
3Reliability
If canonicalization is performed on all occurrences of each value, then complete data type coverage is achieved, but redundant processing increases time consumption
Solution Approach 1:
The patent segments the processing task by first identifying distinct values through aggregation, then applying canonicalization and type-dependent processing only to those distinct values. This segmentation ensures complete data type coverage for all unique values while eliminating redundant processing of duplicate occurrences, thereby improving processing efficiency without sacrificing reliability.
Solution Approach 2:
The system performs preliminary aggregation to identify the set of distinct values before performing canonicalization. This preliminary action ensures that all unique data types are captured for reliable profiling while avoiding the excessive time consumption that would result from processing every occurrence of each value multiple times.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Processing data in a computing system includes receiving a plurality of records that each have one or more values for respective fields of a plurality of fields. Data type information associates each of one or more data types with at least one identifier. Processing a plurality of data values from the records includes: generating a plurality of data units from the records, each data unit including a field identifier that uniquely identifies one of the fields and a binary value from one of the records, the binary value extracted from the field of that record identified by the field identifier; aggregating information about binary values from a plurality of the data units; generating a list of entries for each of one or more of the fields, at least some of the entries each including one of the binary values and information about that binary value aggregated from a plurality of the data units; retrieving a data type associated with a first identifier from the data type information, and associating the retrieved data type with at least one binary value included in an entry of one of the lists; and generating profile information for at least one of the fields based at least in part on a retrieved data type of a particular binary value appearing in the field, after aggregating information about binary values from a plurality of the data units.