Language Model Document Clustering for Small Dataset Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional clustering methods for datasets, particularly with small datasets, often result in poor cohesion of clusters, require separate algorithms for generating cluster titles, and fail to match cluster content, while traditional ML algorithms perform poorly with fewer data points, leaving many un-clustered.
Innovation Solution
A method using a first language model for clustering and a second language model for classification, with iterative processes and batching to manage context token capacity, allowing for hierarchical and overlapping clusters, and intruder detection to ensure quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If traditional clustering methods are used, then clustering can be performed, but the clusters are not cohesive and require separate algorithms for generating cluster titles
Solution Approach 1:
The patent combines multiple clustering algorithms (hierarchical clustering, DBSCAN, K-means) into a single integrated system that performs both clustering and title generation simultaneously. The language model generates coherent cluster titles as an intrinsic part of the clustering process, eliminating the need for separate title generation algorithms and improving overall system reliability.
Solution Approach 2:
The system employs a universal language model that performs multiple functions: it clusters documents, generates cluster titles, and evaluates cluster quality. This multi-functional approach replaces traditional specialized algorithms for each task, improving ease of manufacture while maintaining high reliability through the model's comprehensive capabilities.
2Productivity
If traditional ML algorithms are used with small datasets, then processing can be performed, but performance declines and many data points remain un-clustered
Solution Approach 1:
The patent changes the fundamental parameters of the clustering approach by using a language model with adjustable context window sizes and iterative processing. This allows the system to adapt to small datasets by processing documents in batches and refining clusters iteratively, achieving high productivity (95% classification rate) while maintaining reliability on small datasets where traditional algorithms fail.
Solution Approach 2:
The system dynamically adjusts its processing approach based on dataset characteristics. It uses iterative clustering where the language model processes documents in multiple passes, refining cluster assignments and generating titles dynamically. This dynamic approach enables high productivity on small datasets by efficiently utilizing the limited data points without leaving them un-clustered.
3Reliability
If language models with large context windows are used for clustering, then high-quality clustering is achieved, but computational resources and time increase
Solution Approach 1:
The patent segments the clustering process into iterative batches, where the language model processes documents in manageable chunks rather than all at once. Each iteration refines cluster assignments and generates titles for processed documents. This segmentation maintains high clustering quality by allowing the model to focus on smaller subsets, reducing the effective context window burden and processing time while preserving reliability.
Solution Approach 2:
The system employs periodic action through iterative clustering passes. The language model performs multiple iterations of clustering and title generation, with each pass improving cluster quality. This periodic approach balances computational resources by distributing the heavy processing load across multiple time intervals rather than requiring all resources simultaneously, reducing overall processing time while maintaining high reliability.
Data Source
AI summary
Improved solutions for dataset clustering and evaluation are disclosed. Examples cluster a set of documents into set of clusters using a language model, in an iterative process. In second and later clustering tasks, the current cluster titles and descriptions are provided in the language model prompt, to avoid near-duplications. Upon determining that the set of clusters is sufficiently complete and representative of the set of documents, the tasking switches to classification of the set of documents into the set of clusters using a language model. Classification continues until a sufficient percentage of the set of documents is classified. Some examples use batching, to avoid overloading the language model(s). In some examples, different language models are used for clustering and classification. Some examples use intruder detection to determine the quality of the clustering. This process provides superior performance on classifying documents having little configuration control, such as website feedback from consumers.