Weak-Supervised Information Retrieval Model Training for Long Documents
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Artificial intelligence-based information retrieval models require large amounts of labeled data for training, which is uneconomical, and often struggle with processing long documents due to limitations in the length of documents they can handle, such as the 512-token limit in BERT models.
Innovation Solution
The method involves generating pseudo-labels and training an AI-based IR model using weak supervision by creating pseudo-queries from documents, forming pseudo-query-document relationships, and training the model with these pseudo-labels, allowing for search functionality even with unlabeled data and long documents by dividing documents into passages and using unsupervised learning for initial retrieval and re-ranking.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If labeled data is used for training AI-based IR models, then model accuracy is improved, but labeling cost increases significantly
Solution Approach 1:
The system generates its own training data by automatically creating pseudo-labels through unsupervised learning on the corpus itself. The IR model extracts candidate passages and generates pseudo-labels without human intervention, making the training process self-sufficient and eliminating labeling costs while maintaining model accuracy
Solution Approach 2:
The system creates synthetic training examples by copying and adapting the structure of existing search systems (BM25, TF-IDF) to generate pseudo-relevant passages. These synthetic copies serve as training data, replacing expensive human-labeled data while preserving the essential training signal
2Loss of information
If AI-based IR models process long documents, then information retrieval completeness is improved, but processing capability is limited by token constraints
Solution Approach 1:
The patent divides long documents into multiple candidate passages of manageable length (e.g., 512 tokens each). The system segments the document corpus into passage units that fit within model processing constraints, then uses unsupervised learning to identify and combine relevant segments, maintaining information completeness while respecting processing limitations
Solution Approach 2:
The system transitions from processing documents as single monolithic units to processing them as collections of passage segments. By adding the dimensional aspect of passage segmentation and recombination, the system can handle arbitrarily long documents by breaking them into manageable pieces and reconstructing relevance through multiple passages
3Ease of operation
If unsupervised learning is used for information retrieval, then ease of operation is improved, but search accuracy deteriorates compared to AI-based models
Solution Approach 1:
The system performs preliminary unsupervised learning to generate candidate passages and pseudo-labels before final model training. This preliminary action creates a foundation of candidate results that are then refined through the trained model, combining the ease of unsupervised operation with the accuracy of supervised learning outcomes
Solution Approach 2:
The patent introduces pseudo-labels as an intermediary between unsupervised candidate generation and supervised model training. The pseudo-labels act as a bridge, translating unsupervised extraction results into a format suitable for training accurate models, thereby mediating between the simplicity of unsupervised methods and the precision of supervised approaches
Data Source
AI summary
A search method using an artificial intelligence based information retrieval model and a method for training the artificial intelligence based information retrieval model used for the method are provided.In the method, even if there is no labeled data and only a corpus exists, the artificial intelligence based information retrieval model can be trained using the weak-supervision methodology. Search can be performed by dividing documents into passages having short lengths. Compared to an information retrieval model based on unsupervised learning, improved search results are provided.