Tokenization Substring Segmentation Storage Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional tokenization methods require large database tables for long strings, leading to high storage costs and inefficiencies, making it impractical to support a large number of tokenized strings and transactions efficiently.
Innovation Solution
The method partitions non-tokenized strings into smaller substrings, which are used as indices into lookup tables, reducing table sizes and enabling more efficient storage and processing, allowing for real-time transactions with lower costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional tokenization is used for long strings, then complete coverage of all possible values is achieved, but storage requirements become extremely large and cost prohibitive
Solution Approach 1:
The patent divides a long string into multiple smaller substrings and creates separate lookup tables for each substring. For example, a 16-digit number is split into four 4-digit substrings, requiring four lookup tables of size 10,000 each, totaling 40,000 entries instead of 10^16 entries. This segmentation dramatically reduces storage requirements while maintaining complete tokenization coverage.
2Adaptability or versatility
If lookup tables are made larger to support longer strings, then all possible values can be tokenized, but hardware storage costs become enormous
Solution Approach 1:
Instead of creating one large lookup table for all possible 16-digit numbers (10^16 entries), the system segments the problem into four separate lookup tables for 4-digit substrings (10,000 entries each). This allows the system to support the same range of values with dramatically reduced hardware storage costs.
Solution Approach 2:
The patent transforms the problem from a single-dimension lookup (one large table) to a multi-dimension approach (multiple smaller tables indexed by different substrings). This dimensional change allows the system to achieve the same adaptability with reduced storage costs by distributing the lookup across multiple tables.
3Ease of manufacture
If traditional tokenization methods are used, then simplicity of implementation is maintained, but efficiency in handling large numbers of transactions is reduced
Solution Approach 1:
The patent maintains implementation simplicity by using the same basic lookup table structure as traditional tokenization, but applies it to segmented substrings. This allows the system to process multiple transactions more efficiently by reducing the time required to search and retrieve tokens, while the implementation remains conceptually simple and familiar to those experienced with traditional tokenization.
Data Source
AI summary
A non-tokenized string is received. For example, a non-tokenized string could be a credit card number. The non-tokenized string is partitioned into a plurality of non-tokenized substrings. For example, if the credit card number is 16 digits long, it may be partitioned into substrings that are three, six, and seven digits in length. The non-tokenized substrings are used as an index into a plurality of lookup tables. As a result of the indexing, a plurality of tokenized substrings are retrieved. The plurality of tokenized substrings are combined into a tokenized string. The tokenized string is used as a token that represents the credit card number without disclosing the actual credit card number. The reverse of the above process can also occur.