Adaptive Document Chunking with Cosine Similarity

Solution Overview

Problem

Existing AI models struggle with document chunking, leading to resource-intensive searches and incorrect answers due to chunking methods that either lose interrelationships between chunks or incorrectly identify topic changes, resulting in erroneous responses.

Innovation Solution

Implement a document parsing method that segments text into chunks based on common topics using sentence-by-sentence cosine similarity, with adjustable thresholds to manage chunk size and rechunking when necessary, ensuring a balanced distribution of chunk sizes.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Use of energy by moving object

If a large document is chunked into a few large chunks, then the searching may be resource intensive, but the interrelationships between chunks are preserved

Engineering Contradiction:

Improvesearching resource intensityVSAvoidanswer accuracy

Core Design Contradiction:

Use of energy by moving objectVSReliability

Solution Approach 1:

The document is segmented into multiple chunks based on topic similarity using cosine similarity calculations. This segmentation approach creates a balanced chunking strategy that divides the document into manageable pieces while preserving topic coherence within each chunk, thereby reducing search resource intensity without sacrificing answer accuracy.

Inventive Principle:

Principle #1Segmentation

2Use of energy by moving object

If a large document is chunked into a large number of small chunks, then the searching is less resource intensive, but the interrelationships between chunks are lost

Engineering Contradiction:

Improvesearching resource intensityVSAvoidinterrelationships between chunks

Core Design Contradiction:

Use of energy by moving objectVSLoss of information

Solution Approach 1:

The document is segmented into multiple chunks based on topic similarity using cosine similarity calculations. This segmentation approach creates a balanced chunking strategy that divides the document into manageable pieces while preserving topic coherence within each chunk, thereby reducing search resource intensity without sacrificing answer accuracy.

Inventive Principle:

Principle #1Segmentation

3Ease of manufacture

If chunking follows formatting elements such as paragraphs and sections, then the chunking is simple to implement, but topic changes within documents are not detected

Engineering Contradiction:

Improvechunking implementation simplicityVSAvoidtopic change detection accuracy

Core Design Contradiction:

Ease of manufactureVSMeasurement precision

Solution Approach 1:

The patent replaces the mechanical/format-based chunking system (paragraphs, sections) with a semantic-based system using cosine similarity calculations. This substitution enables the detection of topic changes by comparing the semantic similarity between consecutive sentences, allowing the system to identify when a new topic begins even if it occurs within existing formatting structures.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

4Ease of manufacture

If word count-based chunking is used (e.g., every 100 words), then the chunking is easy to implement, but topic coherence within chunks is not maintained

Engineering Contradiction:

Improvechunking implementation simplicityVSAvoidtopic coherence

Core Design Contradiction:

Ease of manufactureVSReliability

Solution Approach 1:

The patent replaces the mechanical/format-based chunking system (paragraphs, sections) with a semantic-based system using cosine similarity calculations. This substitution enables the detection of topic changes by comparing the semantic similarity between consecutive sentences, allowing the system to identify when a new topic begins even if it occurs within existing formatting structures.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Applied Scientific Principles

This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.

Function Achieved in This Case

Improves the accuracy of AI responses by maintaining logical chunking that reflects topic changes, reducing errors and hallucinations, and optimizing resource usage.

Implementation Method 1

sentence-by-sentence cosine similarity test may reveal a number of sentences have a high degree of commonality

Methodology Applied

Scientific EffectCosine similarity:

Data Source

PatentUS20250322165A1Adaptive chunking

Publication Date: 2025.10.16 MICRO FOCUS LLC

AI summary

Systems and methods are provided wherein document content is parsed into sentences, or other structures, and evaluated for similarity to other sentences. The similarity, such as when a consign similarity of two sentences is determined to be greater than a first threshold, is used to place similar sentences into chunks. The sentences are evaluated again based on a second threshold more restrictive than the first threshold. The threshold value is selected to produce a sufficiently flattened distribution of the sizes of the chunks. The sentences may then be re-chunked, and the chunks are then provided to an artificially intelligent language model, such as a large language model (LLM).