Offset Mapping for Wordbreak Tokenization Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Wordbreak algorithms for multi-byte languages like Chinese, Korean, and Japanese often generate processed strings with different lengths and missing characters compared to the original strings, leading to incorrect identification of sensitive information in Data Loss Prevention systems, resulting in potential data theft or accidental disclosure.
Innovation Solution
A computer system that performs a wordbreak algorithm, generates a processed string with tokenized words, and creates an offset map to accurately identify target characters in the original string, ensuring correct classification and protection of sensitive information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a wordbreak algorithm is applied to multi-byte language strings to tokenize and separate words, then word classification and sensitivity detection capability is improved, but character consistency and string integrity are lost due to different lengths and missing characters in processed strings
Solution Approach 1:
The patent introduces an offset map as an intermediary data structure that bridges the original string and the processed tokenized string. This offset map records the positional correspondence between characters in the original string and tokens in the processed string, enabling accurate mapping back to the original character positions even after tokenization transformations. This resolves the contradiction by maintaining character consistency information through the intermediary offset map while still allowing wordbreak processing to improve classification accuracy.
2Productivity
If tokenization is performed on original strings to generate processed strings with separated words, then searchability and classification efficiency are improved, but the processed string length differs from the original string causing identification errors
Solution Approach 1:
The patent implements a feedback mechanism where the offset map provides positional feedback information from the processed string back to the original string. When classification identifies target tokens in the processed string, the offset map feeds back the corresponding positions in the original string, enabling accurate identification of target characters despite the length differences caused by tokenization. This feedback loop resolves the contradiction by maintaining identification accuracy while preserving classification efficiency.
Data Source
AI summary
A computer system is provided, including a processor coupled to a mass storage device that stores instructions, which, upon execution by the processor, cause the processor to store an original string formed of a plurality of characters, perform a wordbreak algorithm on the original string, and tokenize the original string to generate a processed string including a plurality of word tokens separated by spaces. The processor is further configured to generate an offset map between locations within the word tokens in the processed string and corresponding locations in the original string and classify a portion of the processed string as a target. The processor is further configured to identify target characters in the original string that correspond to the target using the offset map and perform a predetermined action on the target characters in the original string.