In-Database Predictive Modeling Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional predictive modeling techniques face performance and scalability limitations due to the need to transfer large datasets across networks for processing, consuming bandwidth and requiring capable hardware, which restricts their ability to handle growing throughput and large datasets effectively.
Innovation Solution
In-database predictive modeling is performed on a distributed processing platform like Apache Hadoop, where data-intensive steps are delegated to the platform, reducing network bandwidth usage and leveraging its processing capabilities, and sensitive data is securely handled without insecure communication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional predictive modeling techniques transfer large datasets across networks for processing, then the models can be trained on comprehensive data, but network bandwidth is consumed and hardware requirements increase
Solution Approach 1:
The patent introduces an intermediary mechanism where the predictive modeling system directly accesses and processes data within the database management system using SQL-based operations, rather than transferring data across the network. The model training occurs through database-native operations that compute statistics and patterns in-place, eliminating the need to move large datasets across network boundaries while still enabling comprehensive data analysis
Solution Approach 2:
The database management system performs predictive modeling operations autonomously using its own processing capabilities and storage resources. The system leverages existing database infrastructure to execute model training directly on stored data without requiring external computational resources or data extraction, enabling the database to serve its own analytical needs efficiently
2Quantity of substance
If traditional predictive modeling transfers large datasets for processing, then comprehensive analysis is possible, but hardware capabilities must be increased
Solution Approach 1:
The patent uses the database management system as an intermediary that provides the computational infrastructure needed for predictive modeling. Instead of requiring separate high-capacity hardware to handle data transfer and processing, the system leverages the database's existing storage and processing capabilities to perform model training directly on the stored data
Solution Approach 2:
The database management system performs multiple functions: it stores data, processes queries, and executes predictive modeling operations. By making the database multi-functional, the system eliminates the need for separate specialized hardware for data transfer and model training, reducing overall hardware requirements while maintaining comprehensive data analysis capabilities
3Productivity
If data is transferred across networks for predictive modeling, then model training can be performed, but security risks increase due to insecure communication
Solution Approach 1:
The database management system acts as a secure intermediary that enables model training without exposing data to insecure network communication. The system provides controlled access to data through SQL operations and performs computing tasks within the secure database environment, eliminating the need to transfer sensitive data across potentially vulnerable network channels while still enabling comprehensive model training
Data Source
AI summary
Techniques are described for identifying an input training dataset stored within an underlying data platform; and transmitting instructions to the data platform, the instructions being executable by the data platform to train a predictive model based on the input training dataset by delegating one or more data processing operations to a plurality of nodes across the data platform.