String Data Pre-processing for Network Transmission
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large string files, such as those containing gigabytes of genetic sequence data, pose challenges for network transmission due to their size, leading to bottlenecks and increased latency, with conventional compression techniques only offering a 75% reduction in file size, which is insufficient to address these issues.
Innovation Solution
The system employs pre-processing techniques that extract sequences from string files, generate run-length encodings, and utilize hash tables to combine and cluster encoded sequences, resulting in a 1000:1 reduction in file size, thereby reducing network resource consumption and transmission time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional compression techniques are used to reduce string data size, then some reduction is achieved (75%), but the reduction is insufficient for very large string files and network transmission efficiency remains poor
Solution Approach 1:
The patent segments the string data into fixed-width fields based on a defined schema, extracting only the necessary data elements for transmission. This segmentation allows the system to eliminate redundant information and transmit only essential data, achieving up to 1000:1 size reduction while maintaining data integrity and enabling efficient network transmission.
Solution Approach 2:
The system extracts specific data fields from the string data according to a predefined schema, removing unnecessary portions of the data. This extraction process isolates only the critical information needed for the intended purpose, dramatically reducing the data volume while preserving essential content, thereby resolving the contradiction between file size reduction and transmission efficiency.
2Reliability
If the original string data is transmitted without pre-processing, then complete data is sent, but network resources are excessively consumed and transmission time increases
Solution Approach 1:
The patent applies preliminary pre-processing actions to the string data before transmission, including schema-based field extraction, validation, and formatting. This preliminary action ensures that only necessary and valid data is prepared for transmission in advance, reducing network resource consumption while maintaining data completeness and reliability through systematic data preparation.
Solution Approach 2:
The system changes the parameter representation of the data by converting unstructured string data into a structured format with defined fields and data types. This parameter transformation enables more efficient encoding and transmission, reducing network resource consumption while preserving the essential information content through optimized data representation.
3Quantity of substance
If compression techniques are applied to reduce file size, then transmission bandwidth is reduced, but the compression is not suitable for very large string files and achieves only limited reduction
Solution Approach 1:
The patent segments very large string files into manageable fixed-width fields according to a schema, enabling systematic processing of large datasets. This segmentation approach is specifically adapted for large files, allowing the system to handle gigabyte-scale data by breaking it into structured, extractable units that can be efficiently processed and transmitted, achieving scalability that conventional compression cannot provide.
Solution Approach 2:
The system creates a simplified copy of the essential data elements from the original large string files, extracting only the necessary fields into a compact format. This copying mechanism generates a reduced representation that captures the essential information while eliminating redundancy, making the solution specifically suitable for very large files where conventional compression methods fail to achieve adequate reduction.
Data Source
AI summary
Systems and methods for pre-processing string data for network transmission are disclosed. A system can extract first sequences from a sequence file and generate respective encoded sequences based on the first sequences extracted from the sequence file. The system can generate a hash table that stores the respective encoded sequences. The system can combine at least two entries in the hash table based on a comparison of data generated from at least two of the respective plurality of encoded sequences. The system can transmit an output file including a plurality of decoded sequences generated based on the hash table.