Parallel Unicode Tokenization Pipelines for Distributed Security
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
There is a need to protect Unicode data in distributed network environments, given the universal adoption of Unicode in information systems and the requirement to account for the format, structure, and characteristics of Unicode data.
Innovation Solution
The solution involves a distributed tokenization environment where a security server converts Unicode characters into code values and instantiates multiple tokenization pipelines for parallel tokenization. Each pipeline performs encoding and tokenization operations, with outputs converted back into characters to form tokenized data, which can be stored or processed remotely.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If Unicode data is protected using traditional tokenization methods, then data security is improved, but processing time and computational resources increase
Solution Approach 1:
The patent divides Unicode data into multiple segments or blocks, each processed by separate tokenization pipelines working in parallel. This segmentation allows the system to maintain high security through comprehensive tokenization while reducing overall processing time by eliminating sequential dependencies between data blocks.
Solution Approach 2:
The system performs preliminary actions by pre-processing Unicode data into code values and organizing it into structured formats before tokenization. This preparation work is done in advance to enable faster parallel processing during the actual tokenization phase, reducing the critical path time for security-critical operations.
2Productivity
If multiple tokenization pipelines are used for parallel processing, then processing efficiency is improved, but system complexity increases
Solution Approach 1:
The patent designs tokenization pipelines with universal, standardized interfaces and common processing logic that can handle different types of Unicode data. This multi-functionality allows multiple pipelines to work in parallel while sharing common resources and coordination mechanisms, reducing the complexity overhead of managing multiple independent systems.
Solution Approach 2:
The system introduces intermediary components such as data distributors and result aggregators that mediate between multiple parallel tokenization pipelines. These intermediaries simplify coordination by providing standardized communication protocols and abstraction layers, reducing the complexity of direct pipeline-to-pipeline interaction.
3Measurement precision
If Unicode data is converted to code values and back, then tokenization accuracy is improved, but processing steps increase
Solution Approach 1:
The conversion of Unicode data to code values is performed as a preliminary action before tokenization. This pre-conversion ensures that the data is in the appropriate format for accurate tokenization processing, while the reversed conversion back to Unicode characters is performed only after tokenization is complete, minimizing the number of conversion steps in the critical path.
Data Source
AI summary
Unicode data can be protected in a distributed tokenization environment. Data to be tokenized can be accessed or received by a security server, which instantiates a number of tokenization pipelines for parallel tokenization of the data. Unicode token tables are accessed by the security server, and each tokenization pipeline uses the accessed token tables to tokenization a portion of the data. Each tokenization pipeline performs a set of encoding or tokenization operations in parallel and based at least in part on a value received from another tokenization pipeline. The outputs of the tokenization pipelines are combined, producing tokenized data, which can be provided to a remote computing system for storage or processing.