Diversity Crawler for Language Model Gap Filling

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing language models face challenges in open domain applications due to the mismatch between training and target contexts, leading to gaps in data representation, especially in dynamic domains like newscasts where current events constantly change, and traditional focused crawling methods are inefficient in collecting diverse web data.

Innovation Solution

A novel crawling strategy that uses an information theoretic measure to guide the crawler towards novelty regions with high perplexity values, avoiding well-represented pages and continuously updating the crawling policy to reflect the current environment, combining techniques from NLP, Machine Learning, and Information Theory to build diverse and generic language models.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If traditional focused crawling methods are used to collect web pages, then the crawler can gather data in well-defined topics, but the language model will have gaps in representing dynamic open domain content and current events

Engineering Contradiction:

Improvelanguage model representation accuracyVSAvoidopen domain adaptability

Core Design Contradiction:

ReliabilityVSAdaptability or versatility

Solution Approach 1:

The crawling policy is made dynamic by continuously updating it based on the current language model's identified gaps. The system adapts to changing open domain content by recalculating perplexity values and adjusting crawling priorities in response to new events and topics, rather than following a static focused crawling approach.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The system uses feedback from the language model's performance metrics (perplexity values) to guide the crawling process. Gaps identified in the language model's representation of open domain content feed back into the crawling policy, directing the crawler to collect pages that address these specific gaps.

Inventive Principle:

Principle #23Feedback

2Reliability

If the crawler collects a large number of web pages to improve language model diversity, then the model coverage increases, but the crawling time and computational resources increase significantly

Engineering Contradiction:

Improvelanguage model diversityVSAvoidcrawling time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system extracts only the specific information needed to fill identified gaps in the language model, rather than collecting comprehensive data across all topics. By focusing crawling efforts on pages with high perplexity values that address specific representation gaps, the system obtains sufficient diversity without requiring exhaustive collection.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The crawling policy parameters are dynamically adjusted based on the current state of the language model. As gaps are filled and the model improves, the crawling strategy adapts by identifying new gaps or adjusting priority weights, allowing efficient resource allocation throughout the iterative process.

Inventive Principle:

Principle #35Parameter changes

3Ease of manufacture

If the crawler focuses on well-represented pages, then the crawling process is simpler, but the language model fails to capture novelty regions and current events

Engineering Contradiction:

Improvecrawling process simplicityVSAvoidnovelty information

Core Design Contradiction:

Ease of manufactureVSLoss of information

Solution Approach 1:

The system introduces perplexity values as an intermediary metric to evaluate and prioritize web pages. This mediator guides the crawler to balance between well-represented pages (simpler to process) and novelty regions (contain important new information), using the perplexity measurement to identify pages that offer the best trade-off.

Inventive Principle:

Principle #24Intermediary (Mediator)

4Adaptability or versatility

If the language model is updated frequently to reflect current events, then the model remains relevant to dynamic content, but the computational cost and processing time increase

Engineering Contradiction:

Improvetemporal adaptabilityVSAvoidcomputational energy

Core Design Contradiction:

Adaptability or versatilityVSUse of energy by moving object

Solution Approach 1:

The language model is updated periodically through iterative crawling cycles rather than continuously. Each cycle collects new pages, updates the model, and recalculates gaps, creating a rhythm of updates that maintains temporal relevance while allowing computational resources to be managed in discrete batches rather than continuous operation.

Inventive Principle:

Principle #19Periodic action

Data Source

PatentUS11328121B2System and method for building diverse language models

Publication Date: 2022.05.10 MICROSOFT TECHNOLOGY LICENSING LLC

AI summary

Disclosed herein are systems, methods, and non-transitory computer-readable storage media for collecting web data in order to create diverse language models. A system configured to practice the method first crawls, such as via a crawler operating on a computing device, a set of documents in a network of interconnected devices according to a visitation policy, wherein the visitation policy is configured to focus on novelty regions for a current language model built from previous crawling cycles by crawling documents whose vocabulary considered likely to fill gaps in the current language model. A language model from a previous cycle can be used to guide the creation of a language model in the following cycle. The novelty regions can include documents with high perplexity values over the current language model.