Continuous Data Profiling Engine Native Database Integration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data profiling methods are inefficient and insecure, requiring export and import of large datasets to third-party platforms, which consumes significant computing resources and poses security risks.
Innovation Solution
Implement a continuous data profiling system that allows profiling to occur natively within the database, using a lightweight CDP manager that communicates directly with the database management system, reducing the need for data export and import.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is exported to third-party platforms for profiling, then data profiling can be performed, but computing resource consumption increases significantly
Solution Approach 1:
The patent merges the data profiling functionality directly into the native database environment by integrating a profiling engine with the database management system. This eliminates the need to export data to external third-party platforms, allowing profiling operations to be performed in-place where the data resides, thereby significantly reducing computing resource consumption associated with data movement and external processing.
Solution Approach 2:
The database management system performs data profiling operations on its own stored data without requiring external assistance. The profiling engine is an integral component that operates autonomously within the database environment, enabling the system to service its own data quality assessment needs without exporting data elsewhere, thus reducing external computing resource dependencies.
2Reliability
If data is exported to third-party platforms for profiling, then data profiling can be performed, but security risks increase due to data mobility
Solution Approach 1:
The patent extracts the data profiling operation from external third-party platforms and relocates it entirely within the native database environment. By taking out the profiling functionality from external systems and embedding it within the database management system, the solution eliminates data mobility to external platforms, thereby removing the security risks associated with exporting and importing data to third-party systems.
Solution Approach 2:
The patent creates a secure, isolated profiling environment within the native database system that acts as an inert atmosphere for data processing. The profiling engine operates in this controlled environment where data never leaves the secure database boundary, protecting it from external security threats, unauthorized access, and unpredictable security measures that characterize third-party platforms.
3Measurement precision
If data is repeatedly exported and imported for profiling, then comprehensive data analysis can be achieved, but time consumption increases
Solution Approach 1:
The patent establishes continuous data profiling capability within the native database environment. The profiling engine can continuously analyze data as it resides in the database without interruption from export-import cycles. This continuous operation maintains comprehensive data analysis capability while eliminating the time losses associated with repeatedly exporting and importing data for profiling operations.
4Reliability
If copies of datasets are used for profiling, then original data remains intact, but additional computing resources are required for reconciliation
Solution Approach 1:
The patent introduces a profiling engine as an intermediary component that operates directly on the native data within the database management system. Rather than creating copies and requiring reconciliation, this intermediary profiling engine analyzes the original data in-place, generating profiling results without modifying the source data. This eliminates the need for complex reconciliation processes while maintaining data integrity through the intermediary's non-intrusive analysis approach.
Data Source
AI summary
The present disclosure is directed to continuous data profiling (CDP). Entities may house large amounts of disorganized and/or duplicative data. To organize and standardize data across a data set, the data may be profiled. However, profiling large data set can be inefficient and give rise to security problems, as profiling datasets typically requires exporting a dataset to a third-party profiling runtime environment. To remedy these issues, the present disclosure is directed to a continuous data profiling platform that comprises a CDP manager communicatively coupled to a client's database. The CDP manager provides access to a CDP API that may install CDP tools on a client's native database environment, enabling the database management system to profile datasets within the client's native database environment, which results in a more efficient use of computing resources and more secure process of profiling datasets.