Sensitive Text Hash Encoding for Low-Collision Data Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing hashing algorithms for data processing functions face challenges in maintaining data security and anonymity while minimizing collisions and performance overheads, particularly when handling sensitive user data.
Innovation Solution
A method involving tokenization, determination of a reduced vocabulary size, and generation of a reference set of hash values to encode sensitive text statements, using a hashing algorithm with a number of bins exceeding the vocabulary size, and training a data processing function on these encoded versions to minimize collisions and reduce memory footprint.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a big hash value space is used to minimize collisions, then collision risk is reduced, but memory footprint and processing overhead increase
Solution Approach 1:
The patent extracts only the essential hash values from the complete hash space by identifying and storing a reference set of hash values corresponding to actual vocabulary tokens. This reduces the memory footprint from the full hash space size to only the necessary portion, while maintaining collision resistance for the actual data being processed.
Solution Approach 2:
The patent performs preliminary action by pre-computing and storing a reference set of hash values during system initialization or training phase. This reference set is then used during actual processing to quickly determine collisions without needing to maintain the entire hash space in memory, thereby reducing ongoing memory requirements and processing overhead.
2Reliability
If hashing algorithm with large number of bins is used, then collision risk is reduced, but processing overhead increases
Solution Approach 1:
The patent extracts and stores only the reference set of hash values that are actually needed for the specific vocabulary being processed, rather than maintaining the complete hash space. This extraction reduces the amount of data that needs to be processed during runtime, thereby reducing processing overhead while maintaining the collision resistance properties of the larger hash space.
3Measurement precision
If sensitive data is stored and transmitted in original form, then processing accuracy is maintained, but security and anonymity are compromised
Solution Approach 1:
The patent introduces hash values as an intermediary representation of the sensitive data. Instead of storing or transmitting the original sensitive text, the system converts it to hash values that serve as a mediator - preserving the ability to perform processing operations and detect collisions while eliminating the exposure of the actual sensitive information. This intermediary layer maintains processing functionality while ensuring security and anonymity.
Data Source
AI summary
An apparatus, program, and method provide data processing functions for processing text statements. A training phase of the data processing function includes: tokenizing each of a plurality of training text statements into respective training sets; determining, among the training sets, the vocabulary size, and a reference set of text tokens, being a subset of the unique text tokens; generating a reference set of hash values by hashing each of the reference set; for each tokenized training text statement, producing an encoded version comprising: for each text token that hashes to one of the reference set of hash values, the hash value; training the data processing function based on the encoded versions of the tokenized text statements. A live phase of the data processing function includes: producing an encoded version of the tokenized input text statement; executing the trained data processing function on the encoded version of the input text statement.