Variable-Length Data Serialization for Disk and Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage and retrieval methods, which involve padding data with leading zeros, result in bloated data sizes, increasing disk usage and network latency, and are inefficient due to fixed binary encoding that loses precision for large numbers.
Innovation Solution
A system and method that serialize customer data by formatting it into first and second data strings, flipping the sign bit, and encoding every two digits into one byte, with additional bit flipping for negative values, to store and retrieve data in a more efficient variable-length format.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If data is padded with leading zeros to maintain fixed binary encoding length, then data storage structure is simplified and retrieval is straightforward, but disk usage increases and data precision for large numbers is lost
Solution Approach 1:
The patent extracts and removes the unnecessary leading zeros from fixed-length binary encoding, retaining only the significant digits. This extraction principle reduces the data volume stored in databases while maintaining the ability to accurately represent numerical values, directly addressing the disk usage problem without sacrificing retrieval simplicity through the use of serialization/deserialization processes.
Solution Approach 2:
The patent transitions from static fixed-length binary encoding to dynamic variable-length encoding based on the actual number of significant digits. This dynamic approach allows the data structure to adapt to the actual content, storing only necessary digits rather than predetermined fixed lengths, thereby reducing disk usage while maintaining retrieval efficiency through structured serialization.
2Ease of manufacture
If data is padded with leading zeros to maintain fixed binary encoding length, then data storage structure is simplified and retrieval is straightforward, but network latency increases due to larger data sizes
Solution Approach 1:
The patent extracts and removes unnecessary leading zeros from the binary encoding, retaining only significant digits. This reduction in data size directly decreases the volume of data transmitted over the network, thereby reducing network latency while maintaining structured storage and retrieval through serialization processes.
Solution Approach 2:
The patent implements dynamic variable-length encoding that adapts to the actual numerical value, storing only the necessary number of digits. This dynamic approach reduces the average data transmission size compared to static fixed-length encoding, directly addressing network latency reduction while preserving data integrity through structured serialization and deserialization.
3Ease of manufacture
If fixed-length binary encoding is used to simplify storage, then storage structure is straightforward, but data precision for large numbers is lost
Solution Approach 1:
The patent extracts and preserves all significant digits without arbitrary truncation imposed by fixed-length constraints. By removing leading zeros and retaining only necessary digits, the system maintains full precision for large numbers while achieving a simplified variable-length storage structure that adapts to the actual data requirements.
Solution Approach 2:
The patent transitions from rigid fixed-length encoding to flexible variable-length encoding that dynamically adjusts to the precision requirements of each numerical value. This dynamic structure allows large numbers to be represented with their full precision by allocating sufficient storage space based on the actual number of significant digits, rather than being constrained by predetermined fixed lengths.
4Ease of operation
If traditional iteration through entire array is used for data search, then search method is simple, but search efficiency is low
Solution Approach 1:
The patent replaces the mechanical iteration process (sequential scanning through the entire array) with a more efficient search mechanism enabled by structured serialization. The serialized format with clear delimiters and patterns allows for optimized search algorithms that can quickly locate specific data without examining every element, thereby improving search efficiency while maintaining operational simplicity through the structured data organization.
Data Source
AI summary
Disclosed embodiments provide systems and methods for reducing disk storage and network latency. A method reducing disk storage and network latency comprises receiving customer data of a customer to store in a database, conditioning the customer data, and formatting the conditioned customer data into first and second data strings respectively having a first data type and a second data type. The method further comprises flipping a sign bit of the first data string, encoding the sign-bit-flipped first data string and second data string into serialized data by representing every two digits of the first string with one byte, and flipping all bits of the serialized data if the received customer data is represented by a negative value. The method further comprises storing the serialized data in the database if negative, receiving a request for the customer data, deserializing serialized data to be retrieved from the database, and retrieving the deserialized data from the database.