Encoded String Processing With Shadow Arrays for Invalid Code Units
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing character encoding schemes, such as UTF-8, face challenges in processing strings with invalid code units, leading to loss of logical positional information and potential security vulnerabilities, as they either discard entire strings or remove positional information to preserve valid data.
Innovation Solution
Generating a shadow array that retains the positional information of both valid and invalid code units, allowing for the reconstruction of the original string, while also creating a processed string that maintains logical order and includes invalid units, thus preserving recoverable information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If existing character encoding schemes process strings with invalid code units by discarding them or removing positional information, then data validity is improved, but logical positional information is lost
Solution Approach 1:
The patent segments the string processing into two parallel outputs: a processed string containing only valid code units, and a shadow array preserving positional information for both valid and invalid code units. This segmentation allows the system to maintain data validity in the processed string while preserving positional information in the shadow array, resolving the contradiction between reliability and information loss.
Solution Approach 2:
The shadow array serves as an intermediary data structure that bridges the gap between the original string and the processed string. It maintains positional correspondence with the original string including invalid code units, while the processed string contains only valid units. This intermediary allows reconstruction of the original string's logical structure without compromising data validity.
2Reliability
If existing character encoding schemes discard entire strings containing invalid code units, then data integrity is improved, but recoverable information is lost
Solution Approach 1:
The patent extracts valid code units from the original string to create the processed string, while separately tracking their positions and the positions of invalid code units in the shadow array. This extraction approach allows the system to maintain data integrity by working with only valid units, while preserving recoverable information about the original string's structure and invalid unit locations.
Solution Approach 2:
The patent changes the parameter representation by introducing a shadow array that stores positional metadata rather than the actual code unit values. This parameter transformation allows the system to work with validated data while maintaining information about the original string's composition, enabling reconstruction without compromising integrity.
3Reliability
If existing character encoding schemes remove positional information to preserve valid data, then data validity is improved, but reconstruction capability deteriorates
Solution Approach 1:
The patent adds another dimension to the data structure by creating a shadow array that exists parallel to the processed string. This shadow array stores positional information in a separate dimension, allowing the system to maintain data validity in the processed string while preserving reconstruction capability in the shadow array without increasing complexity within the same dimension.
Data Source
AI summary
A pointer is set to a first code unit of an original string that encodes characters via code units within an encoding scheme. Whether the code unit of the original string referenced by the pointer is valid within the encoding scheme is determined. If the code unit referenced by the pointer is valid, one or more code units of the original string that encode a single character within the encoding scheme are processed, starting at the code unit referenced by the pointer. The one or more code units as have been processed are appended to a processed string. A single shadow unit indicating that the one or more code units that have been processed are valid is appended to a shadow array. The pointer is advanced to the code unit of the original string following the one or more code units.