Prime N-Gram Representation for Fast Large-Scale String Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing computer architectures are inefficient and resource-intensive when performing string searches in large databases of long strings, particularly in natural language domains, due to issues with character repetition, large database sizes, and resource utilization, leading to slow and inaccurate searches.
Innovation Solution
A computer architecture utilizing a prime representation data structure where each record is a product of selected prime factors associated with n-grams of a domain string, enabling efficient and accurate searches by transforming domain representations into a unique format that allows rapid comparison and division operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional string search algorithms are used on general purpose computers, then the search can be performed on any system, but the search speed and efficiency deteriorate significantly
Solution Approach 1:
The patent segments the string search problem by dividing long strings into n-gram components (contiguous sequences of characters). Each n-gram is assigned a unique prime factor, and the product of these prime factors creates a compact numerical representation. This segmentation transforms the complex string matching problem into efficient numerical operations, resolving the contradiction between versatility and search speed.
Solution Approach 2:
The patent replaces traditional mechanical string comparison operations with mathematical operations. Instead of character-by-character comparison, the system uses prime factorization and modular arithmetic. The search algorithm determines if a short string's prime representation divides evenly into a long string's prime representation, substituting slow mechanical comparison with fast mathematical division and modulo operations.
2Reliability
If character repetition is handled by traditional search engines, then all occurrences are detected, but false matches increase and accuracy deteriorates
Solution Approach 1:
The patent changes the parameter representation from raw characters to prime factor products. By representing each n-gram as a unique prime factor and the entire string as a product of these factors, the system transforms the problem space. This parameter change eliminates false matches caused by character repetition because the prime factorization uniquely represents each n-gram sequence, resolving the contradiction between comprehensive detection and search accuracy.
3Reliability
If large databases of long strings are searched using conventional methods, then complete coverage is achieved, but resource consumption and time requirements increase excessively
Solution Approach 1:
The patent performs preliminary action by pre-computing the prime factor representations for all long strings in the database during an offline phase. This preprocessing step transforms the raw string data into compact numerical form, storing only the essential prime factor products. During online search, the system only needs to perform fast modular arithmetic operations rather than scanning entire strings, dramatically reducing search time while maintaining complete coverage.
Solution Approach 2:
The patent creates a compressed copy of the original string data in prime factor representation. Instead of storing and processing the actual character sequences, the system works with numerical products of prime factors. This copying approach preserves all necessary information for exact matching while reducing data size and processing requirements, resolving the contradiction between search completeness and time efficiency.
4Ease of operation
If simple string operations are performed on long strings, then the operations are easy to implement, but resource usage and computational cost increase significantly
Solution Approach 1:
The patent changes the operational parameters from character manipulation to numerical computation. Basic string operations like equality testing, substring detection, and pattern matching are transformed into arithmetic operations on prime factor products. These mathematical operations are computationally cheaper and more energy-efficient than traditional string handling, while maintaining operational simplicity through consistent mathematical rules.
Data Source
AI summary
An embodiment of the present invention is a prime representation data structure in a computer architecture. The prime representation data structure has a plurality of records where each record contains a prime representation and where the prime representation is a product of two or more selected prime factors. Each of the selected prime factor associated with an n-gram of a domain representation of a domain string. The domain representation of the domain string is a domain string of ordered, contiguous domain characters. The n-gram being a subset of n number of the ordered, contiguous domain characters in the domain string. The computer architecture performs string searching and includes one or more central processing units (CPUs) with one or more operating systems, one or more input/output device interfaces, one or more memories, and one or more input/output devices. The architecture further includes the prime representation data structure, one or more prime target query data structures and a search process performed by one or more of the CPUs. The CPUs can be organized in a hierarchical structure. The prime target query data structure has one or more target prime queries. Each target prime query is the product of one or more target selected prime factors. Each target selected factor is associated with a target n-gram of a target domain representation of a target domain string. The search process, performed by one or more of the CPUs, determines whether one or more of the target selected prime factors is common with one of the selected prime factors. By performing this efficient testing, the computer system can determine if one or more small strings are included in one or more large strings.