Text Embedding Generation in Cloud Enterprise Databases
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
There is no existing solution for scalable text embedding generation and downstream modeling functionalities on cloud enterprise database systems.
Innovation Solution
A computer-implemented method and system for text embedding generation and applications, which allows users to select from multiple text embedding models and generate text embeddings for data elements stored in a data warehouse, enabling downstream machine learning model training and prediction tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If text embedding generation is performed outside the cloud enterprise database system, then text embedding functionality can be achieved, but data migration is required and system complexity increases
Solution Approach 1:
The patent merges text embedding generation functionality directly into the cloud enterprise database system by integrating multiple text embedding models (BERT, RoBERTa, DistilBERT, GPT-2) within the database architecture. This allows users to generate text embeddings for data elements stored in the data warehouse without migrating data externally, thereby simplifying the system architecture and ease of operation while maintaining embedding generation capabilities
2Adaptability or versatility
If multiple text embedding models are provided, then model selection flexibility improves, but model selection complexity increases
Solution Approach 1:
The patent implements a self-service mechanism where the system automatically selects an appropriate text embedding model based on query parameters such as cost thresholds and quality thresholds. The system evaluates multiple models (BERT, RoBERTa, DistilBERT, GPT-2) and autonomously determines the optimal model to use, eliminating the need for manual model selection and reducing management complexity while maintaining high adaptability
3Quantity of substance
If text embedding generation is performed on large datasets, then embedding coverage improves, but processing time increases
Solution Approach 1:
The patent enables efficient processing of large datasets by allowing dynamic adjustment of generation parameters including batch size, number of workers, and memory allocation. The system can process embeddings for entire data warehouses by optimizing these parameters, achieving high embedding coverage while minimizing processing time through parallel processing and resource allocation strategies
Data Source
AI summary
A method includes receiving a text embedding generation query from a user requesting generation of a text embedding for one or more data elements stored at a data warehouse. In response, the method includes selecting, using the text embedding generation query, a text embedding model from a plurality of different text embedding models. The method includes generating, using the selected text embedding model, the text embedding for the one or more data elements and storing the text embeddings at the data warehouse. The method includes receiving a machine learning model training query from the user device requesting training of a machine learning model using the text embeddings. In response to receiving the machine learning model training query, the method includes training the machine learning model using the text embeddings. The method includes providing, to the user device, a notification indicating that training of the machine learning model is complete.