Unicode Dictionary Compression Using Private Use Code Points
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
There is a mismatch in data representation between programming languages like C and JavaScript, particularly in handling character strings, leading to issues during compression where character codes can span octets, causing errors in displayed strings.
Innovation Solution
A method and apparatus for transcoding Unicode character data using a compression algorithm that assigns encoded values to private use code points in the Unicode format, employing techniques like LZW and LZSS with UTF encoding to ensure compatibility and efficiency across different character sets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If compression algorithms are used to reduce data size, then storage efficiency is improved, but data integrity and compatibility across different programming languages may deteriorate
Solution Approach 1:
The patent introduces Unicode as an intermediary representation system that mediates between different character encoding schemes (such as ASCII and other character sets). By converting all character data to Unicode code points before compression, the system ensures that data maintains its semantic meaning across different programming languages and platforms while still achieving compression benefits.
Solution Approach 2:
The patent transforms character data from various encoding schemes into a standardized parameter system (Unicode code points U+0000 to U+FFFF). This parameter transformation allows the compression algorithm to work with a consistent data representation, ensuring that compressed data can be reliably decompressed and displayed regardless of the original encoding or the programming language used.
2Productivity
If character codes are compressed by spanning octets, then compression ratio is improved, but compatibility with languages expecting one character per octet deteriorates
Solution Approach 1:
The patent segments the Unicode code point space into distinct ranges, with the Private Use Area (U+E000 to U+F8FF) specifically allocated for compression dictionary entries. This segmentation allows the compression algorithm to use multi-byte representations efficiently while reserving other segments for standard character representations that maintain compatibility with languages expecting one character per byte.
Solution Approach 2:
The patent transitions from a one-dimensional byte-based representation to a two-dimensional representation where the first dimension is the Unicode code point value and the second dimension is the encoding format (UTF-8, UTF-16, etc.). This dimensional change allows the system to achieve better compression ratios by utilizing the full range of Unicode code points while maintaining compatibility through proper encoding selection.
3Measurement precision
If Unicode code points are used for compression, then character representation accuracy is improved, but memory requirements for decompression increase
Solution Approach 1:
The patent applies partial action by using only the necessary portion of the Unicode space (Basic Multilingual Plane, U+0000 to U+FFFF) for compression dictionary entries, rather than the full Unicode range. This partial utilization of the code space achieves sufficient character representation accuracy for most practical applications while keeping memory requirements manageable.
Solution Approach 2:
The patent implements dynamic memory management where the compression dictionary is built adaptively during the compression process. Frequently occurring character sequences are added to the dictionary and assigned Private Use Area code points, while less frequent sequences remain in the standard Unicode ranges. This dynamic approach optimizes the balance between representation accuracy and memory usage based on the actual data being compressed.
Data Source
AI summary
A character data set is compressed with a compression algorithm module of a computer system to generate one or more streams of encoded values. A code point mapper assigns the encoded values to code points reserved for private use in a Unicode format. An encoder encodes the streams of assigned encoded values using a Unicode Transformation Format (UTF). A dictionary of the compression algorithm initially includes code points assigned by the Unicode format. New entries are successively assigned to private use code points.