Script Similarity Search Using LLM Embeddings for New Content
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing similar content searches, particularly for dramas and scripts, face limitations in accurately assessing similarity based solely on textual content, leading to inaccurate search results.
Innovation Solution
A method involving script data preprocessing, episode-level and content-level featurization using pre-trained large language models (PLLM), Doc2Vec, and TF-IDF, followed by similarity score determination through cosine similarity and weighted averaging, to enhance the accuracy of content similarity searches.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If collaborative filtering model is used for similar content search, then user behavior data can be leveraged for recommendations, but accuracy deteriorates when dealing with new dramas or scripts that lack sufficient behavioral data
Solution Approach 1:
The patent replaces the collaborative filtering approach (which relies on user behavior data) with a content-based similarity search system that uses text embedding models. This substitution enables the system to assess similarity based on the actual textual content of scripts rather than derivative information, thereby improving accuracy for new content while maintaining adaptability.
Solution Approach 2:
The patent introduces text embedding vectors as an intermediary representation between the script content and the similarity assessment. By converting script text into embedding vectors using pre-trained language models, the system creates a meaningful intermediate representation that captures semantic information, enabling accurate similarity comparison even for new dramas or scripts without sufficient behavioral data.
2Measurement precision
If search is conducted based on textual content with supplementary elements (genre, setting, period, theme), then broader coverage is achieved, but accuracy deteriorates due to insufficient textual content analysis
Solution Approach 1:
The patent applies preliminary action by using pre-trained language models that have already learned semantic representations from large corpora. These pre-trained models perform the heavy lifting of understanding language nuances before the actual similarity search, allowing the system to achieve high accuracy without increasing the complexity of the search process itself. The preprocessing step converts text into meaningful embeddings that capture semantic information.
Solution Approach 2:
The patent extracts the essential semantic information from script text by converting it into embedding vectors. This extraction process separates the meaningful content representation from the raw text and supplementary elements, allowing the system to focus on the core textual content for similarity assessment while ignoring less relevant metadata.
3Productivity
If script data is divided into multiple chunks for embedding, then processing efficiency is improved, but information loss may occur during chunking
Solution Approach 1:
The patent applies segmentation by dividing long script texts into smaller chunks that can be processed by the embedding model. This segmentation enables efficient processing of large volumes of text while maintaining the ability to capture semantic information within each chunk. The system processes multiple chunks and aggregates their representations to form a comprehensive understanding of the entire script.
Data Source
AI summary
The present invention relates to a method, system, and program for searching similar content based on large language models. Specifically, it pertains to a method, system, and program that allow users to search for similar content based on script texts and to compare and analyze the content.