Distributed Data Profiling for Source-Level Validation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems fail to efficiently validate data for its intended purpose, leading to unnecessary network bandwidth and storage consumption when data does not meet specific criteria, particularly in scenarios where different use cases have varying data validation requirements.
Innovation Solution
Implementing a data profiling and validation system that evaluates data at the source level using distributed processes, generating profile data and validation rules to ensure data meets criteria before loading it into tables, thereby avoiding unnecessary data transfer and storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If data is transferred and stored without prior validation, then data availability is maintained, but network bandwidth and storage resources are wasted
Solution Approach 1:
The patent performs data profiling and validation at the source system before data transfer, evaluating data characteristics and validating against usage requirements in advance. This preliminary action identifies unsuitable data early, preventing unnecessary network transfer and storage consumption while ensuring only appropriate data is moved.
Solution Approach 2:
The patent introduces a data profiling and validation intermediary layer between the source system and destination system. This intermediary evaluates data characteristics and validates suitability for intended usage, acting as a gatekeeper that prevents inappropriate data from consuming network bandwidth and storage resources.
2Loss of energy
If data is validated at the destination system, then data suitability is ensured, but network bandwidth and storage resources are consumed
Solution Approach 1:
The patent performs data profiling and validation at the source system before data transfer, evaluating data characteristics and validating against usage requirements in advance. This preliminary action identifies unsuitable data early, preventing unnecessary network transfer and storage consumption while ensuring only appropriate data is moved.
Solution Approach 2:
The patent enables the source system to perform self-validation of its own data against specified usage requirements. The source system evaluates its data characteristics and determines suitability for intended purposes without requiring complex validation infrastructure at the destination system.
3Reliability
If different validation criteria are applied for different use cases, then data suitability is improved, but validation process complexity increases
Solution Approach 1:
The patent applies different validation criteria locally for different data usage scenarios. Each data usage requirement has its own specific validation rules tailored to that purpose, allowing appropriate rigor for each use case without requiring a single complex universal validation system.
Solution Approach 2:
The patent creates a universal data profiling and validation framework that can handle multiple different validation criteria through a single system. The validation engine can evaluate data against various usage requirements using different criteria sets, providing multi-functional capability without proportionally increasing complexity.
Data Source
AI summary
Disclosed are various approaches performing distributed profiling and validation. First, a request to load data stored in a data warehouse into a table in a data store is received. Then, a determination as to whether the data stored in the data warehouse complies with a validation rule is made, wherein the validation rule is based at least in part on profile data for the data stored in the data warehouse, and wherein the validation rule is associated with an intended use for the data stored in the data warehouse. Then, in response to a determination that the data store in the data warehouse does not comply with the validation rule, a notification is sent to a predefined recipient.