Deep Learning Text Similarity Filtering via Feature Vector Comparison
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge lies in efficiently filtering out similar information from vast amounts of data, such as text or images, as existing methods are inefficient and labor-intensive when comparing information one by one.
Innovation Solution
A deep learning-based method that involves acquiring text data, processing it through a deep learning model to calculate feature vectors, and comparing these vectors to determine similarity, with the computing apparatus recording new text data only when the similarity is below a predetermined threshold, thereby filtering out similar information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If text data is collected by comparing information one by one, then filtering accuracy is improved, but collection efficiency deteriorates
Solution Approach 1:
The patent segments text data into fixed-length sequences (e.g., 5 tokens) and processes them in batches rather than comparing individual texts one by one. This segmentation enables parallel processing of multiple text sequences simultaneously, maintaining filtering accuracy while significantly improving collection efficiency.
Solution Approach 2:
The patent replaces the mechanical one-by-one comparison system with a deep learning-based automated system. The computing apparatus uses neural networks to automatically compare and filter text data, substituting manual or sequential mechanical comparison with intelligent automated processing that achieves both high accuracy and efficiency.
2Productivity
If deep learning model is used to determine similarity, then filtering efficiency is improved, but computational complexity increases
Solution Approach 1:
The patent applies local quality by processing text data in fixed-length segments (local portions) rather than analyzing entire texts globally. Each segment is processed independently through the deep learning model, reducing the computational burden on each processing unit while maintaining overall filtering efficiency through parallel processing of multiple segments.
Data Source
AI summary
According to the present invention, provided is a method for collecting filtered text data, comprising the steps in which: a computing device acquires first text data and records the first text data in a text data pool; the computing device acquires second text data; the computing device performs a calculation in a deep learning model by using the first text data and the second text data as input values and calculates a first feature vector corresponding to the first text data and a second feature vector corresponding to the second text data; and the computing device compares the degree of similarity between the first feature vector and the second feature vector, and records the second text data in the text data pool when the degree of similarity is less than a predetermined value.