Tokenized Bigram Search for Secure Distributed Data Retrieval

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Data stored in a database becomes unsearchable after encoding, posing a security risk when decoded, and exact search results are limited due to misspelled query terms or term variations.

Innovation Solution

Tokenize database entries and bigrams of each data entry, allowing query terms to be parsed into bigrams for matching tokenized database entries without decoding, enabling secure and accurate searching.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If data is encoded to protect it, then security is improved, but the data becomes unsearchable

Engineering Contradiction:

Improvedata securityVSAvoidsearchability

Core Design Contradiction:

ReliabilityVSEase of operation

Solution Approach 1:

The patent segments data into bigrams (pairs of consecutive characters) and tokenizes each bigram separately. This segmentation allows the system to search for partial matches and handle misspelled queries while maintaining security, as the tokenized bigrams can be searched without decoding the full data entries.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces tokenized bigrams as an intermediary layer between the encoded data and search queries. These tokenized bigrams serve as searchable metadata that mediates between the security requirement (encoded data) and the search requirement (queryable data), enabling searches without exposing or decoding sensitive information.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Measurement precision

If exact matching is used for search accuracy, then precision is improved, but similar but not exact results are missed due to misspelled query terms

Engineering Contradiction:

Improvesearch precisionVSAvoidsearch flexibility

Core Design Contradiction:

Measurement precisionVSAdaptability or versatility

Solution Approach 1:

The patent applies partial matching by searching for bigrams that partially overlap with query terms. Instead of requiring exact full-term matches, the system searches for individual bigrams within the data, allowing misspelled or variant query terms to still match relevant data entries through partial bigram overlaps.

Inventive Principle:

Principle #16Partial or excessive action

Solution Approach 2:

By segmenting both data and queries into bigrams, the system enables flexible partial matching. This segmentation allows the search to identify relevant entries even when query terms are misspelled or differ slightly, as long as some bigrams match between the query and the data.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS20250371191A1Data tokenization and search in a distributed network

Publication Date: 2025.12.04 PROTEGRITY US HLDG LLC

AI summary

Data in a database can be protected, for instance by tokenizing the entries of the database using one or more token tables. To enable searching data within the database without first detokenizing the tokenized database entries, bigrams of each data entry can also be tokenized and stored in association with the tokenized data entry. When a query term is received, the query term can be parsed into bigrams, and each bigram can be tokenized. The tokenized query bigrams can be used to query the database, and tokenized database entries corresponding to tokenized bigrams that match the tokenized query bigrams can be identified and returned as search results.