Distributed Data Profiler Using Spark In-Memory Processing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing data profiling tools face performance issues when dealing with large datasets, often taking hours or days to generate comprehensive statistics, and may fail to profile multi-terabyte tables efficiently, especially with MapReduce approaches.

Innovation Solution

A method and system utilizing Apache Spark and Scala to profile datasets by querying metadata storage, allocating resources, shuffling and repartitioning data blocks, and computing aggregates across a distributed computing system, enabling simultaneous generation of extensive statistical metrics.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Quantity of substance

If traditional MapReduce approach is used to profile large datasets, then the system can handle big data, but the profiling time becomes very long (hours or days)

Engineering Contradiction:
Improvedataset sizeVSAvoidprofiling time
Core Design Contradiction:
Quantity of substanceVSLoss of time

Solution Approach 1:

The patent changes the computational parameters by switching from MapReduce to Apache Spark, utilizing in-memory processing capabilities to dramatically reduce profiling time while maintaining the ability to handle large datasets

Inventive Principle:
Principle #35Parameter changes

Solution Approach 2:

The patent replaces the mechanical MapReduce processing system with a Spark-based system that uses in-memory computation, thereby reducing the time required for profiling operations on large datasets

Inventive Principle:
Principle #28Mechanics substitution (Replace mechanical system)

2Quantity of substance

If MapReduce approach is used for computation intensive metrics, then the system can process data, but it fails to generate extensive statistic metrics on multi-terabyte tables

Engineering Contradiction:
Improvedata volumeVSAvoidmetric generation capability
Core Design Contradiction:
Quantity of substanceVSProductivity

Solution Approach 1:

The patent changes the processing paradigm from disk-based MapReduce to in-memory Spark processing, enabling the system to generate extensive statistical metrics on multi-terabyte tables that were previously unachievable with traditional MapReduce

Inventive Principle:
Principle #35Parameter changes

3Measurement precision

If existing profiling tools are used, then some statistics can be obtained, but they take very long time or fail to generate extensive statistic metrics

Engineering Contradiction:
Improvestatistic metricsVSAvoidprofiling time
Core Design Contradiction:
Measurement precisionVSLoss of time

Solution Approach 1:

The patent replaces traditional profiling tools with a Spark-based system that uses in-memory processing, enabling extensive statistic metrics to be generated much faster while maintaining measurement precision

Inventive Principle:
Principle #28Mechanics substitution (Replace mechanical system)

Data Source

PatentUS10521440B2High performance data profiler for big data
Publication Date: 2019.12.31 AETNA INC
  • US10521440B2 patent drawing
  • US10521440B2 patent drawing
  • US10521440B2 patent drawing

AI summary

A method for profiling a dataset includes: querying, by a data profiler executed on a distributed computing system, a metadata storage to obtain table information; allocating, by the data profiler, system resources based on the obtained table information; profiling, by the data profiler, the dataset to obtain profiling results, wherein profiling the dataset includes shuffling and repartitioning data blocks of the dataset with respect to a plurality of nodes of the distributed computing system, and computing aggregates based on the shuffled and repartitioned data blocks; and outputting, by the data profiler, the profiling results.