Pre-Indexing Filter and Corrector for Search Systems

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing search systems face challenges in efficiently indexing diverse and large volumes of documents, leading to increased processor cycles, memory usage, and search time, particularly in contexts like litigation where documents vary significantly in type and size, resulting in less useful indices with extraneous tokens.

Innovation Solution

A pre-indexing filter and corrector system that evaluates documents using detectors, scorers, and judgers to determine suitability for indexing, allowing for context-specific filtering and alteration of documents to reduce the burden of indexing, thereby improving processing efficiency and reducing unnecessary tokens.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If all documents are indexed to ensure completeness, then indexing completeness is improved, but processor cycles and memory usage increase

Engineering Contradiction:

Improveindexing completenessVSAvoidprocessor cycles

Core Design Contradiction:

ReliabilityVSUse of energy by moving object

Solution Approach 1:

The system performs preliminary evaluation of documents using detectors and scorers before indexing. Documents are pre-screened to assess their suitability for indexing based on content characteristics, allowing the system to make informed decisions about which documents warrant full indexing processing and which can be excluded, thereby reducing overall processor cycles while maintaining indexing completeness for relevant documents

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system applies different indexing strategies to different documents based on their individual characteristics. By evaluating each document's content through detectors and assigning scores, the system determines whether to apply full indexing or simplified processing, creating a differentiated indexing approach that optimizes resource usage while maintaining completeness for documents that require it

Inventive Principle:

Principle #3Local quality

2Reliability

If all documents are indexed to ensure completeness, then indexing completeness is improved, but memory usage increases

Engineering Contradiction:

Improveindexing completenessVSAvoidmemory usage

Core Design Contradiction:

ReliabilityVSQuantity of substance

Solution Approach 1:

The system performs preliminary evaluation of documents using detectors and scorers before indexing. Documents are pre-screened to assess their suitability for indexing based on content characteristics, allowing the system to make informed decisions about which documents warrant full indexing processing and which can be excluded, thereby reducing overall memory usage while maintaining indexing completeness for relevant documents

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system applies different indexing strategies to different documents based on their individual characteristics. By evaluating each document's content through detectors and assigning scores, the system determines whether to apply full indexing or simplified processing, creating a differentiated indexing approach that optimizes memory usage while maintaining completeness for documents that require it

Inventive Principle:

Principle #3Local quality

3Reliability

If all documents are indexed to ensure completeness, then indexing completeness is improved, but search time increases

Engineering Contradiction:

Improveindexing completenessVSAvoidsearch time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system performs preliminary evaluation of documents using detectors and scorers before indexing. Documents are pre-screened to assess their suitability for indexing based on content characteristics, allowing the system to make informed decisions about which documents warrant full indexing processing and which can be excluded, thereby reducing overall search time while maintaining indexing completeness for relevant documents

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system applies different indexing strategies to different documents based on their individual characteristics. By evaluating each document's content through detectors and assigning scores, the system determines whether to apply full indexing or simplified processing, creating a differentiated indexing approach that optimizes search time while maintaining completeness for documents that require it

Inventive Principle:

Principle #3Local quality

4Reliability

If documents with extraneous tokens are indexed, then indexing completeness is improved, but the quality of the index decreases

Engineering Contradiction:

Improveindexing completenessVSAvoidindex quality

Core Design Contradiction:

ReliabilityVSLoss of information

Solution Approach 1:

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

Inventive Principle:

Principle #3Local quality

Data Source

PatentUS20240004930A1System and method for pre-indexing filtering and correction of documents in search systems

Publication Date: 2024.01.04 OPEN TEXT HOLDINGS INC

AI summary

Embodiments as disclosed herein provide a search system with an pre-indexing filter that provides both a sophisticated and contextually tailored approach to filtering documents and a corrector that is adapted to alter a document that has been designated to be filtered out from the indexing process and determine if the altered document should be indexed. The alteration of the document may be tied to the attributes, rules or thresholds used to initially filter the document from the indexing process. The filtering criteria can thus be tailored to a specific context such that both the initial filtering and the alteration process may be better suited for application in that context.