Parallel Data Profiling via Vertical Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data profiling methods in parallel processing environments are inefficient, particularly when searching for specific information in columns, as they often retrieve and process entire data sets, leading to resource consumption and inconsistency issues due to dynamic data updates.
Innovation Solution
The method involves vertical partitioning of data, where only the relevant attribute subsets are retrieved and processed on different machines, ensuring consistent results by creating staging files and merging sample results to form a profiled subset, which can be transmitted, displayed, or stored.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all data files are retrieved to profile a single column, then complete profiling information is obtained, but network communication costs and processing time increase significantly
Solution Approach 1:
The patent segments the data architecture into multiple layers: data source layer, projection computer layer, and client layer. Profiling operations are performed on segmented projections (vertical partitions) rather than entire datasets. This allows the system to retrieve only relevant attribute subsets from specific projections, dramatically reducing network communication overhead while maintaining complete profiling capability for the target column.
2Loss of information
If entire data sets are processed for column profiling, then all attribute values are available, but resource consumption increases
Solution Approach 1:
The patent extracts only the necessary attribute subsets required for profiling from specific projections, rather than processing entire datasets. The system identifies which projections contain the target column and retrieves only those relevant attribute subsets, eliminating unnecessary data movement and processing while ensuring complete profiling information is obtained.
3Ease of operation
If data is horizontally partitioned across files, then specific subject lookup is efficient, but column-based profiling requires retrieving all files
Solution Approach 1:
The patent introduces a vertical partitioning dimension complementing the existing horizontal partitioning. Data is organized both by subjects across horizontal files and by attributes across vertical projections. This two-dimensional organization allows profiling operations to exploit the vertical dimension, retrieving only relevant attribute subsets from specific projections rather than scanning all horizontal files, thereby dramatically improving profiling efficiency.
4Reliability
If profiling is performed on the complete data set, then consistent results are ensured, but processing overhead increases due to dynamic data updates
Solution Approach 1:
The patent performs preliminary profiling operations on static projections that are less susceptible to dynamic updates. By profiling attribute subsets from these stable projections and merging results, the system achieves consistent profiling outcomes without requiring complete data set processing. This preliminary action on selected projections reduces processing complexity while maintaining reliability.
Data Source
AI summary
A computer-implemented method of profiling a data set in a parallel processing environment includes vertically partitioning an initial data set. One or more attribute subsets are then profiled. A list of subjects is generated each corresponding to a specific attribute value identified in the profiling. Values of multiple attributes are extracted for each identified subject, and the sample results are assembled and merged.