Binary Stream Encoding for Document Content Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for detecting common content in documents, such as plagiarism and near-duplication, are computationally intensive and brittle, failing to efficiently identify closely-related content and being unsuitable for Cloud/Crowd-based services due to high computational costs and sensitivity to minor textual changes.
Innovation Solution
A binary stream representation of text using minimal encoding with one digit per meaningful character string, allowing for efficient pattern matching and robust identification of closely-related content without disclosing the original content, employing a character string recognizer, encoder, and matcher for document comparison.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional coding mechanisms (Hash or Rabin's integer values) are used for pattern matching, then identical text recognition is effective, but computational cost and time delay increase significantly
Solution Approach 1:
The patent changes the parameter representation from traditional Hash/Rabin codes (32-128 digits) to a minimal binary encoding scheme (1 digit per meaningful character string). This parameter transformation reduces the computational complexity of both coding and pattern matching operations while maintaining the ability to recognize identical text, thereby resolving the contradiction between reliability and productivity.
Solution Approach 2:
The patent extracts only the essential meaningful character strings from text and represents them with minimal binary digits, eliminating redundant information. This extraction approach reduces the data size required for pattern matching from large integer codes to compact binary streams, significantly improving processing speed while preserving identical text recognition capability.
2Measurement precision
If traditional coding mechanisms are used, then identical content can be detected, but the system becomes brittle to minor changes and fails to identify near-identical content
Solution Approach 1:
The patent segments text into meaningful character strings and represents each segment with a minimal binary digit. This segmentation approach allows the system to process text at a granular level, enabling it to detect near-identical content by comparing individual segments independently. Minor changes affect only specific segments rather than the entire content code, improving robustness while maintaining matching precision.
Solution Approach 2:
The patent applies partial matching by allowing comparisons at the segment level rather than requiring complete exact matches of entire content codes. This partial action approach enables the system to identify near-identical content where only some segments match, thereby improving reliability against textual variations while preserving measurement precision for exact matches.
3Reliability
If large integer codes are generated for each text fragment, then pattern matching can identify exact matches, but the computational complexity and memory requirements increase
Solution Approach 1:
The patent creates a simplified binary copy representation of text content instead of using complex integer codes. Each meaningful character string is copied and represented by a single binary digit, creating a compact surrogate that preserves pattern matching capability while dramatically reducing computational complexity and memory requirements for both coding and matching operations.
Data Source
AI summary
A computer-based detection tool for detecting whether content within a given document is common to content within a plurality of other, existing documents, the detection tool comprising: a character string recognizer for recognizing character strings in the content of the given document; a character string distinguisher for distinguishing main character strings and auxiliary character strings in the recognized character strings by reference to a closed list of main character strings; an encoder for encoding the content of the given document by assigning one or more digits to each main character string and one or more digits to auxiliary character strings; and a matcher for matching a plurality of n-digit streams from within the encoded content with any corresponding n-digit streams within previously-encoded content of the one or more other documents. The character strings may be encoded as a bit-stream.