Stateless Tokenization Using Static Token Tables for Fast Detokenization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional stateful tokenization systems require maintaining a database of clear text and token pairs, leading to increased processing and storage burdens, potential single points of failure, and high costs due to the ever-growing database size.
Innovation Solution
Implementing a stateless tokenization system that uses a limited static token table in computer memory for tokenization and detokenization, eliminating the need for a database and enabling distributed processing across multiple systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If stateful tokenization systems maintain a database of clear text and token pairs, then de-tokenization can be achieved by lookup, but processing and storage burdens increase monotonically over time
Solution Approach 1:
The patent pre-generates a complete token table mapping all possible clear text values to tokens before runtime. This preliminary action eliminates the need for database lookups during tokenization and de-tokenization operations, as all mappings are already available in memory, thus resolving the contradiction between maintaining de-tokenization capability and preserving processing speed.
Solution Approach 2:
The patent creates a static copy of the token mapping table in computer memory rather than maintaining a dynamic database. This copying approach allows the system to have immediate access to all token mappings without the overhead of database queries, solving the contradiction by having the complete mapping available locally without the burden of maintaining a growing database.
2Reliability
If stateful tokenization systems maintain a database of clear text and token pairs, then de-tokenization can be achieved by lookup, but storage burden increases monotonically over time
Solution Approach 1:
The patent pre-generates a complete token table mapping all possible clear text values to tokens before runtime. This preliminary action eliminates the need for database lookups during tokenization and de-tokenization operations, as all mappings are already available in memory, thus resolving the contradiction between maintaining de-tokenization capability and preserving processing speed.
Solution Approach 2:
The patent creates a static copy of the token mapping table in computer memory rather than maintaining a dynamic database. This copying approach allows the system to have immediate access to all token mappings without the overhead of database queries, solving the contradiction by having the complete mapping available locally without the burden of maintaining a growing database.
3Reliability
If stateful tokenization systems maintain a database of clear text and token pairs, then de-tokenization can be achieved by lookup, but security requirements and expense increase
Solution Approach 1:
The patent extracts the token mapping functionality from a centralized database and implements it as a stateless algorithm using pre-generated token tables in memory. This extraction eliminates the security vulnerabilities associated with maintaining a centralized database of clear text-token pairs, as no sensitive data needs to be stored or transmitted, resolving the contradiction between maintaining de-tokenization capability and reducing security risks.
Solution Approach 2:
The patent uses ephemeral in-memory token tables that are generated once and discarded after use, rather than maintaining persistent database connections. This approach minimizes security exposure by ensuring that token mappings exist only temporarily in memory and are not stored on disk or in databases, eliminating the security risks associated with database maintenance while preserving de-tokenization capability.
4Reliability
If stateful tokenization systems maintain a database of clear text and token pairs, then de-tokenization can be achieved by lookup, but system performance degrades
Solution Approach 1:
The patent pre-generates a complete token table mapping all possible clear text values to tokens before runtime. This preliminary action eliminates the need for database lookups during tokenization and de-tokenization operations, as all mappings are already available in memory, thus resolving the contradiction between maintaining de-tokenization capability and preserving processing speed.
Solution Approach 2:
The patent creates a static copy of the token mapping table in computer memory rather than maintaining a dynamic database. This copying approach allows the system to have immediate access to all token mappings without the overhead of database queries, solving the contradiction by having the complete mapping available locally without the burden of maintaining a growing database.
Data Source
Figure 1
Figure 2
Figure 3A~3E
AI summary
A method for securing data by stateless tokenization includes receiving a character string, selecting a character window within the character string, accessing a token table at an index equal to a value of the character window, retrieving, from the token table at the index, a tokenized value, replacing the value of the character window with the tokenized value, shifting the character window by one character in a first direction, accessing the token table at the index equal to the value of the character window, retrieving a tokenized value, replacing the value of the character window with the tokenized value, shifting the character window by one character in a second direction, accessing the token table at the index equal to the value of the character window, retrieving a tokenized value, replacing the value of the character window with the tokenized value, and returning the character string as a tokenized character string.