SQL Data Pre-Processing for Secure In-Database ML Preparation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data pre-processing operations in machine learning are computationally expensive and time-consuming due to processing large amounts of data stored in not-so-optimized structures, and moving data jeopardizes security and privacy.
Innovation Solution
Perform data pre-processing operations directly on the database where the data is stored, leveraging SQL-based database servers for enhanced performance and security by generating and executing SQL statements for data pre-processing functions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data pre-processing is performed using traditional machine learning libraries, then data can be transformed and prepared for modeling, but the process becomes computationally expensive and time-consuming
Solution Approach 1:
The patent translates and copies the functionality of machine learning data pre-processing libraries into SQL implementations within the database system. Instead of executing complex pre-processing operations in application code, equivalent SQL statements are generated and executed directly in the database, leveraging the database engine's optimized query processing capabilities to achieve faster execution with lower computational overhead.
Solution Approach 2:
The patent introduces an intermediary translation layer that converts machine learning library function calls into SQL statements. This intermediary module acts as a bridge between the machine learning workflow and the database system, enabling pre-processing operations to be executed efficiently within the database environment rather than requiring data movement and external processing.
2Ease of operation
If data is moved from storage to processing systems for pre-processing, then data transformation can be performed, but data security and privacy are jeopardized
Solution Approach 1:
The patent introduces an intermediary translation layer that converts machine learning library function calls into SQL statements. This intermediary module acts as a bridge between the machine learning workflow and the database system, enabling pre-processing operations to be executed efficiently within the database environment rather than requiring data movement and external processing.
Solution Approach 2:
The patent translates and copies the functionality of machine learning data pre-processing libraries into SQL implementations within the database system. Instead of executing complex pre-processing operations in application code, equivalent SQL statements are generated and executed directly in the database, leveraging the database engine's optimized query processing capabilities to achieve faster execution with lower computational overhead.
3Manufacturing precision
If extensive data is processed during the training stage, then comprehensive pre-processing can be achieved, but the training stage becomes lengthy and computationally expensive
Solution Approach 1:
The patent translates and copies the functionality of machine learning data pre-processing libraries into SQL implementations within the database system. Instead of executing complex pre-processing operations in application code, equivalent SQL statements are generated and executed directly in the database, leveraging the database engine's optimized query processing capabilities to achieve faster execution with lower computational overhead.
Data Source
AI summary
In accordance with an embodiment of the invention, a method is provided for performing data pre-processing operations during data preparation of a machine learning lifecycle. The method includes defining one or more data pre-processing functions for applying to data stored in a dataset, executing one or more learn functions for learning the data, and executing one or more transform functions for transforming the data. Each of the one or more learn functions generates a first Structured Query Language (SQL) statement representing a definition of corresponding learn function for corresponding defined data pre-processing function. Each of the one or more transform functions generates a second SQL statement representing a definition of corresponding transform function for corresponding defined data pre-processing function. The dataset is stored in a database.