Source Data Profiling for Pre-Transfer Validation in Warehouses
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data validation methods are inefficient and wasteful, as they often require transferring large datasets across networks for profiling, consuming bandwidth and storage without ensuring data fitness for its intended purpose.
Innovation Solution
Implementing a system where data profiling is performed at the source database level, using distributed processes to generate profile data and validation rules, which are then used to determine data fitness before transfer, thereby avoiding unfit data transmission.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is transferred across networks for profiling, then data validation can be performed, but network bandwidth and storage are consumed without ensuring data fitness
Solution Approach 1:
The system performs data profiling at the source database before data transfer occurs. Validation rules are evaluated against profile data generated at the source, determining data fitness for purpose in advance. This preliminary validation prevents unnecessary transfer of unfit data, conserving network bandwidth while ensuring reliability of transferred data.
Solution Approach 2:
The invention extracts only the necessary profile data and validation results from the source database before transfer. Instead of transferring entire datasets for validation, only essential metadata and profile information are extracted and sent to the validation system, significantly reducing network bandwidth consumption while maintaining validation effectiveness.
2Reliability
If data is transferred across networks for profiling, then data validation can be performed, but storage resources are wasted on unfit data
Solution Approach 1:
The system performs data profiling and validation assessment before data transfer. By evaluating profile data against validation rules at the source, the system determines which datasets are fit for their intended purpose in advance, preventing storage of unfit data and optimizing storage resource utilization.
Solution Approach 2:
The invention extracts only validated, fit-for-purpose data for storage and transfer. By filtering data at the source based on validation results, only necessary and useful data is transferred and stored, eliminating waste of storage capacity on unfit or inappropriate data.
3Reliability
If data profiling is performed centrally, then validation can be done, but processing time increases and throughput decreases
Solution Approach 1:
The system segments the data validation process by distributing validation rules to multiple source databases. Each source database independently evaluates its own profile data against relevant validation rules, enabling parallel processing of multiple datasets simultaneously. This segmentation maintains validation accuracy while significantly increasing overall profiling throughput.
Solution Approach 2:
Source databases perform self-validation by evaluating their own profile data against distributed validation rules. This self-service approach eliminates the need for centralized sequential processing, allowing each database to validate its data independently and in parallel, thereby maintaining reliability while improving productivity.
Data Source
AI summary
Disclosed are various approaches performing distributed profiling and validation. A query is generated for profile data for data stored in a data warehouse. Then, the query for the profile data is sent to the data warehouse. The profile data is then received from the data warehouse. Later, in response to a request to load the data stored in the data warehouse into a table in a data store, a determination is made as to whether the data stored in the data warehouse complies with a validation rule.