Adaptive Serialized Data Slicing for Low-Bandwidth Transmission
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud computing platforms face inefficiencies in data transmission due to large data volumes, leading to high electricity consumption, environmental impact, and vulnerability to cyber-attacks, with existing solutions requiring costly and risky reengineering.
Innovation Solution
A method and system for optimizing data transmission using dynamic and adaptive slicing and reduction of serialized data through a mutable and polymorphic data reduction matrix, which reduces data size without losing information, applicable to various data formats and transmission channels.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If data is transmitted in traditional serialized formats (JSON, XML, HTML), then data is human understandable and self-descriptive, but data volume is large leading to high bandwidth consumption and electricity usage
Solution Approach 1:
The patent segments serialized data into fixed-size blocks (e.g., 64KB chunks) for processing. Each block is independently compressed and transmitted, allowing efficient bandwidth utilization while maintaining data integrity. This segmentation enables parallel processing and reduces the memory footprint required for handling large datasets.
Solution Approach 2:
The patent transforms data from traditional serialized formats (JSON, XML, HTML) into a binary format with changed parameters: fixed block sizes, compact encoding schemes, and optimized data structures. This parameter transformation reduces data volume by eliminating redundancy while maintaining essential information, thereby reducing bandwidth consumption without completely sacrificing human readability through structured formatting.
2Reliability
If data is transmitted in traditional serialized formats, then data maintains integrity and structure, but data volume increases leading to higher electricity consumption in data centers
Solution Approach 1:
The patent applies preliminary compression and blocking to data before transmission. By pre-processing data into compact binary blocks with embedded metadata, the system reduces the storage space required and consequently lowers electricity consumption in data centers. Data integrity is preserved through structured block formats that include validation information.
Solution Approach 2:
The patent creates a compact binary representation (copy) of the original serialized data. This copied format maintains all essential information and structural integrity while occupying significantly less storage space. The binary copy can be seamlessly converted back to readable formats when needed, ensuring data reliability while reducing storage energy requirements.
3Use of energy by moving object
If existing technologies are converted to binary formats, then bandwidth consumption is reduced, but reengineering efforts and complexity increase
Solution Approach 1:
The patent designs a universal binary data format that can represent multiple data types (strings, numbers, booleans, nested structures) using a single standardized schema. This multi-functional format eliminates the need for separate conversion logic for different data types, reducing reengineering complexity while achieving bandwidth reduction through consistent binary encoding across all data kinds.
Solution Approach 2:
The patent implements dynamic block processing where data is divided into fixed-size blocks that can be independently processed, transmitted, and reassembled. This dynamic approach allows existing systems to adopt the binary format incrementally without complete reengineering, as each block can be handled independently by standard network protocols and the original data structure is preserved through metadata in each block.
4Use of energy by moving object
If data is compressed and transmitted in smaller formats, then bandwidth consumption is reduced, but data processing and decompression time increases
Solution Approach 1:
The patent segments data into fixed-size blocks that can be processed independently and in parallel. This segmentation reduces the processing time for each individual block compared to processing entire large datasets sequentially. The fixed block size optimizes memory usage and enables efficient cache utilization during decompression, balancing compression benefits with processing speed.
Solution Approach 2:
The patent uses parameter-optimized binary encoding where common data types use compact representations with fewer bits. This parameter optimization reduces the amount of data to be transmitted and processed, offsetting the decompression overhead. The binary format includes type indicators that allow receiving systems to quickly identify and process different data kinds using appropriate decoding routines, minimizing processing time.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
There is provided a system and a method for dynamically and adaptively slicing, reducing and bit packing a streamed serialized data sequence. A streamed serialized data sequence is received, and a reduction matrix is iteratively applied on the streamed serialized data sequence to obtain a plurality of sliced values. For a given iteration, it is determined if a corresponding given sliced value of the plurality of sliced values is present in an index. If the given sliced value is not present, the given sliced value is added by bit packing an unused id associated with the given sliced value using a first number of bits based on a current index length and using a second number of bits to encode the sliced data value to thereby obtain a given bit packed sliced value. The given bit packed sliced value is aggregated to obtain a bit packed data stream and transmitted.