Sensitive Text Hash Encoding for Low-Collision Data Processing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing hashing algorithms used in data processing functions face challenges with collisions, leading to accuracy issues and performance overheads due to large hash value spaces, which compromise the security and efficiency of processing sensitive data.

Innovation Solution

A method involving tokenization, determination of a reduced reference set of hash values, and encoding of input text statements using a hashing algorithm with a number of bins exceeding the vocabulary size, followed by training and executing the data processing function on encoded versions, thereby reducing the risk of collisions and memory footprint.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If a big hash value space is used to minimize collisions, then collision risk is reduced, but memory footprint and performance overhead increase

Engineering Contradiction:

Improvecollision riskVSAvoidmemory footprint

Core Design Contradiction:

ReliabilityVSQuantity of substance

Solution Approach 1:

The patent extracts only the most frequently occurring tokens from the complete vocabulary to form a reference set. This reference set is then hashed to create a condensed hash table that contains only the essential mappings needed for accurate data processing, eliminating the need to maintain hash entries for all possible tokens in the large vocabulary space.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent changes the parameter of hash table size from being proportional to the complete vocabulary size to being proportional to the size of the reference set (most common tokens). This parameter change dramatically reduces memory footprint while maintaining collision resistance for the dominant portion of input data.

Inventive Principle:

Principle #35Parameter changes

2Measurement precision

If hashing algorithm is configured to have many bins to reduce collisions, then accuracy is improved, but performance overhead increases

Engineering Contradiction:

Improvedata processing accuracyVSAvoidprocessing speed

Core Design Contradiction:

Measurement precisionVSProductivity

Solution Approach 1:

The patent extracts and processes only the reference set of most common tokens through hashing, while less common tokens are handled through alternative mechanisms. This extraction approach maintains high accuracy for frequent cases without incurring the full performance cost of hashing every possible token.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent applies partial hashing action by only hashing the reference set of most common tokens rather than the complete vocabulary. This partial action is sufficient to maintain accuracy for the majority of data processing cases while significantly reducing computational overhead.

Inventive Principle:

Principle #16Partial or excessive action

3Measurement precision

If complete vocabulary is hashed to ensure accuracy, then data processing accuracy is maintained, but memory footprint increases

Engineering Contradiction:

Improvedata processing accuracyVSAvoidmemory footprint

Core Design Contradiction:

Measurement precisionVSQuantity of substance

Solution Approach 1:

Inventive Principle:

Principle #2Taking out (Extraction)

4Ease of operation

If sensitive data is stored and transmitted in plaintext for processing, then data processing functionality is maintained, but security is compromised

Engineering Contradiction:

Improvedata processing functionalityVSAvoiddata exposure risk

Core Design Contradiction:

Ease of operationVSObject-affected harmful factors

Solution Approach 1:

The patent performs preliminary hashing of sensitive data before storage and transmission. By hashing the data in advance, the system enables subsequent data processing operations on the hashed values without needing to expose the original sensitive data, thus maintaining both security and functionality.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent introduces hashed values as an intermediary representation of sensitive data. These hash values serve as mediators that allow data processing functions to operate on the data without directly accessing or exposing the original sensitive information, thereby maintaining security while enabling functionality.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS12547742B2Method, program, and apparatus for processing sensitive data

Publication Date: 2026.02.10 XERO

AI summary

An apparatus, program, and method provide data processing functions for processing text statements. A training phase of the data processing function includes: tokenizing each of a plurality of training text statements into respective training sets; determining, among the training sets, the vocabulary size, and a reference set of text tokens, being a subset of the unique text tokens; generating a reference set of hash values by hashing each of the reference set; for each tokenized training text statement, producing an encoded version comprising: for each text token that hashes to one of the reference set of hash values, the hash value; training the data processing function based on the encoded versions of the tokenized text statements. A live phase of the data processing function includes: producing an encoded version of the tokenized input text statement; executing the trained data processing function on the encoded version of the input text statement.