LLM-Based User Embedding Generation for Unstructured Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Online systems face significant resource expenditure in generating training data for embedding models, particularly when dealing with unstructured data like free text, which requires conversion to structured data for effective user and content embeddings.
Innovation Solution
An online system utilizes a large language model (LLM) to generate user embeddings based on user interaction data, eliminating the need for structured input data by capturing nuanced connections within textual information and incorporating interaction timing for scaling or weighting item embeddings, and applies a transformer network to produce a final user embedding for personalized item selection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional embedding models are used to generate user embeddings, then structured data can be processed effectively, but significant resources must be expended converting unstructured data to structured data
Solution Approach 1:
The patent changes the fundamental parameter of data structure requirement from structured to unstructured by adopting LLM-based embedding generation. The system accepts unstructured user interaction data (free text, logs, events) directly without requiring conversion to structured formats, thereby eliminating the resource-intensive data preprocessing step while maintaining or improving embedding quality through the LLM's advanced language understanding capabilities
Solution Approach 2:
The patent replaces the mechanical data conversion process (transforming unstructured data into structured data through parsing, cleaning, and formatting operations) with a direct LLM-based embedding generation mechanism. The LLM inherently understands and processes unstructured text data, substituting the multi-step mechanical conversion pipeline with a single neural network-based embedding generation step, thereby reducing computational resources and processing time
2Loss of energy
If LLMs are used to generate embeddings from unstructured data, then resource expenditure for data conversion is reduced, but the complexity of processing large amounts of text data increases
Solution Approach 1:
The patent applies preliminary action by pre-training the LLM on vast amounts of text data before deployment. The LLM is trained in advance to understand language patterns, semantics, and contextual relationships, so that during actual embedding generation, it can directly process unstructured user interaction data without requiring complex real-time processing pipelines. This shifts complexity from runtime to training time, simplifying the operational system
Solution Approach 2:
The patent introduces the LLM as an intermediary between unstructured user interaction data and the embedding space. Instead of directly converting unstructured data to structured formats or using multiple processing stages, the LLM serves as a single powerful intermediary that ingests raw text data and outputs high-quality embeddings, thereby simplifying the overall system architecture despite the LLM's internal complexity
3Productivity
If traditional embedding models require structured input data, then the inference process is efficient, but the system cannot effectively utilize free text user interaction data
Solution Approach 1:
The patent applies universality by making the embedding generation system capable of handling multiple data formats (unstructured text, structured data, logs, events) through a single LLM-based approach. The LLM can process various types of user interaction data in their native formats without requiring conversion to a specific structured schema, thereby achieving both flexibility in data acceptance and efficiency in embedding generation through a unified processing pipeline
Data Source
AI summary
An online system selects an item to present to a user of the online system. The online system accesses user interaction data for the user. The online system transmits the user interaction data to a model serving system and receives, from the model serving system, item embeddings for the items with which the user interacted. The model serving system may use an LLM to generate the item embeddings based on the user interaction data. The online system generates a user embedding array based on the item embeddings. The online system applies a transformer network to the user embedding array to generate a user embedding describing the user. To select an item to present to the user, the online system compares the generated user embedding to item embeddings for a set of candidate items. The online system selects a candidate item based on the interaction scores.