Document Node Grouping for Near-Duplicate Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for detecting near duplicate documents in large datasets are inefficient, particularly in litigation proceedings where massive amounts of documents need to be reviewed, and existing systems struggle to accurately identify and group similar documents.
Innovation Solution
A computer-based system that organizes documents into nodes based on their inclusiveness, where documents with substantially identical normalized body text and subject parameters are grouped together, allowing for efficient comparison and identification of near duplicates through fingerprinting and normalization processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all documents are reviewed individually to ensure complete review, then review completeness is improved, but review time and effort increase significantly
Solution Approach 1:
The patent merges documents that have substantially identical normalized body text into the same node, so that reviewing one document represents reviewing all merged documents. This combining approach maintains review completeness while reducing the total number of documents that need to be read, directly resolving the contradiction between review completeness and review time.
Solution Approach 2:
The system performs preliminary normalization and grouping of documents into nodes before the actual review process. By pre-organizing documents with identical content into the same node and identifying inclusive relationships, the system reduces the review workload in advance, allowing reviewers to focus only on unique or more inclusive documents while ensuring comprehensive coverage.
2Measurement precision
If documents are grouped by exact match only, then grouping accuracy is improved, but many near duplicate documents remain ungrouped
Solution Approach 1:
The patent changes the grouping criterion from exact match to 'substantially identical normalized body text'. By normalizing the body text of documents and comparing normalized versions, the system can accurately group near-duplicate documents that differ only in minor formatting or wording variations, thereby improving both grouping accuracy and review efficiency simultaneously.
Solution Approach 2:
The system segments the document comparison process into normalization and comparison stages. By first normalizing the body text of each document and then comparing the normalized versions, the system can identify near-duplicates more accurately while maintaining efficient grouping, resolving the contradiction between precision and productivity.
3Loss of information
If reviewers read all documents to ensure no information is missed, then information completeness is improved, but the screening process becomes tedious and inefficient
Solution Approach 1:
The patent merges documents with substantially identical normalized body text into the same node, allowing reviewers to read one representative document from each node instead of multiple identical documents. This merging approach ensures that no information is lost while dramatically improving screening efficiency and reducing the tedium of reviewing redundant documents.
Solution Approach 2:
The system performs preliminary organization of documents into nodes based on identical normalized body text and identifies inclusive relationships before the review process. This pre-processing ensures information completeness is maintained while making the actual screening process easier and more efficient by presenting reviewers with a condensed, non-redundant set of documents.
4Ease of operation
If a hierarchical node structure is implemented to organize documents, then document organization is improved, but system complexity increases
Solution Approach 1:
The patent segments the document set into a hierarchical tree structure of nodes, where each node contains documents with substantially identical normalized body text. This segmentation organizes documents in a manageable hierarchical format that improves ease of operation while the systematic approach to node creation and linking keeps the overall system complexity controlled.
Solution Approach 2:
The patent implements a nested hierarchical structure where nodes can contain other nodes, creating a tree-like organization. Documents with identical content are grouped in leaf nodes, while parent nodes represent inclusive relationships. This nesting approach provides excellent document organization while following a systematic pattern that manages complexity through regular structural rules.
Data Source
AI summary
A computer product including a data structure for organizing of a plurality of documents, and capable of being utilized by a processor for manipulating data of the data structure and capable of displaying selected data on a display unit. The data structure includes a plurality of directionally interlinked nodes, each node being associated with one or more documents having a header and body text. All the documents are associated with a given node and have identical normalized body text. All documents that have identical normalized body text are associated with the same node. One or more of the nodes is associated with more than one document. For any node that is a descendent of another node, the normalized body text of each document associated with the node is inclusive of the normalized body text of a document that is associated with the other node.