In-Database Computational Programming for Distributed Analytics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems face performance constraints and high resource costs when executing computational procedures that involve large data sets stored in foreign databases, as they require data to be read and written back, leading to expensive operations in terms of time and resources, especially with symmetric multi-processing techniques not effectively reducing network data transmission.
Innovation Solution
The implementation of in-database computational programming techniques that allow analytic code to be executed directly within a distributed database system, using a compute model to convert general requests into database-specific formats and perform calculations across multiple nodes, enabling massive parallel processing and reducing the need for extensive data transfer.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is read into process space before computation, then computation can be performed, but response time increases and resource costs increase
Solution Approach 1:
The patent inverts the conventional computation model by moving the computation engine into the database system itself rather than reading data out to external computation environments. This allows procedures to execute directly against data in the database, eliminating data transfer time and enabling computation to occur where the data resides, thus resolving the contradiction between computation execution and response time
Solution Approach 2:
The patent introduces a procedure execution engine as an intermediary component that bridges the database management system and analytical procedures. This engine enables direct in-database computation by managing procedure execution, parameter binding, and result retrieval within the database environment, eliminating the need to read data into external process space while maintaining full computation capability
2Productivity
If data is read into process space for computation, then analysis can be performed, but disk or memory space requirements increase
Solution Approach 1:
The patent extracts the computation engine from external process space and embeds it directly within the database management system. This extraction eliminates the need to load large datasets into external memory spaces, as computation occurs in-place within the database environment, thus reducing memory space requirements while maintaining analytical computation capability
Solution Approach 2:
The database management system provides self-service computational capabilities by integrating the procedure execution engine directly into the database. This allows the database to perform analytical computations on its own stored data without requiring external computation environments or additional memory spaces, enabling the system to serve its own computational needs
3Speed
If multiple OS threads are used with SMP techniques, then response time may be improved, but network data transmission between client and database remains high
Solution Approach 1:
The patent inverts the client-server computation model by moving the computation engine from the client side into the database server. This inversion eliminates the need for clients to download large datasets for local processing, thereby reducing network data transmission while maintaining parallel processing capabilities through the integrated execution engine that can utilize multiple database threads
Data Source
AI summary
Various embodiments are generally directed to an apparatus and method for generating a general request having structures and information to perform an analytical calculation on data stored in a distributed database system and converting the structures and information of the general request to a compute request having a request format conforming to a query language used by the distributed database system. Various embodiments may also include sending the compute request to a node of the distributed database system and receiving a compute response from the node of the distributed database system, the compute response including a result set of the analytical calculation performed on data local to the node from an analytic container implemented by the node, the analytic container including an embedded process to replicate an execution environment hosted within the distributed database system used by a client application.