Automated Text Fragment Detection Using Shingle Hashing and Inverted Indexes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for detecting confidential data in text documents within internal company networks are time-consuming and have limited applicability, failing to provide a fast and universally applicable solution for identifying phrases, sentences, or text fragments.
Innovation Solution
The method involves converting text documents into a binary stream, extracting meaningful characters, forming 'shingles' by removing punctuation and converting to lowercase, calculating hash-values, and creating an inverted index to efficiently compare and identify matching text fragments between analyzed and model documents, using a coefficient of coincidence equation to determine relevance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional semantic net comparison methods are used to detect confidential data in documents, then measurement precision is improved, but productivity deteriorates due to time-consuming analysis
Solution Approach 1:
The patent segments text documents into smaller units called 'clauses' (sentences or phrases) and further processes them into 'shingles' (n-grams of words). This segmentation allows the system to compare only relevant portions of documents rather than entire documents, significantly improving analysis speed while maintaining detection accuracy through the coefficient of coincidence calculation.
Solution Approach 2:
The patent replaces the mechanical semantic net comparison process with a computational hash-based system. By converting clauses into shingles, calculating their hash values, and using inverted indexes for rapid lookup, the system achieves fast comparison without the computational overhead of building and comparing full semantic nets, thus improving productivity while preserving measurement precision.
2Measurement precision
If comprehensive text analysis methods are applied to all documents, then measurement precision is improved, but loss of time increases due to processing large volumes of data
Solution Approach 1:
The patent performs preliminary actions by pre-processing documents into clauses and shingles, calculating hash values, and building inverted indexes before actual comparison operations. This preliminary structuring allows rapid retrieval and comparison during analysis, reducing processing time for large volumes of data while maintaining comprehensive analysis capabilities for high measurement precision.
Solution Approach 2:
The patent extracts only the essential meaningful elements from documents - specifically clauses and their shingles - rather than analyzing entire documents. By extracting and indexing these key elements, the system achieves comprehensive text analysis with reduced processing time, as only extracted elements need to be compared rather than processing all text data.
3Measurement precision
If detailed clause-by-clause comparison is performed between documents, then measurement precision is improved, but device complexity increases due to multiple processing steps
Solution Approach 1:
The patent introduces intermediary structures - clauses as intermediaries between full documents and shingles, and inverted indexes as intermediaries between shingles and comparison operations. These intermediaries organize data in a way that simplifies the comparison process while maintaining detailed analysis capabilities, thus improving measurement precision without proportionally increasing device complexity.
Solution Approach 2:
The patent transforms text data through parameter changes - converting clauses into shingles by removing punctuation and converting to lowercase, then transforming shingles into hash values. These parameter transformations standardize the data format and enable efficient comparison operations, achieving detailed clause-by-clause analysis with manageable system complexity through consistent data transformation rules.
4Adaptability or versatility
If traditional text scanning methods are used to find matching phrases, then adaptability is improved for different document types, but productivity deteriorates due to limited applicability and slow processing
Solution Approach 1:
The patent creates a universal processing pipeline that handles different document types through standardized steps: conversion to clauses, transformation to shingles, hash calculation, and inverted index creation. This universal approach maintains adaptability to different document types while dramatically improving productivity, as the same efficient algorithmic process applies across all document types without requiring document-specific processing logic.
Data Source
AI summary
Automated analysis of text documents is used to scan text documents in order to find phrases or text fragments from other documents, or modifying the existing ones. A comparatively fast and universally applicable method finds phrases, sentences or even text fragments from other documents. The method includes: all electronic files containing model documents are converted to a given format; meaningful fragments, called “clauses”, are extracted from them; the converted files containing model documents are stored in the database; each electronic file containing a document to be analyzed is converted to the given format; clauses extracted from analyzed documents are compared with clauses extracted from model documents; fractions of clauses from an analyzed document matching clauses from each model document are calculated; fractions found are then compared with a pre-set threshold value in order to find out whether there are text fragments from a model document in the analyzed one.